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Abstract 


The  Performance  Work  Statement  (PWS),  which  defines  the  requirements  and 
shapes  the  outcome  of  the  activity  under  contract,  and  the  Quality  Assurance  Surveillance 
Plan  (QASP),  which  is  vital  in  identifying  what  is  to  be  evaluated,  become  critical 
documents  as  the  Air  Force  enters  into  more  competitive  sourcing  contracts.  Currently, 
contractors  and  Most  Efficient  Organizations  (MEO)  are  evaluated  by  the  QASP  based 
upon  the  requirements  found  in  the  PWS.  It  is  imperative  the  PWS  and  QASP  documents 
have  adequate  performance  metrics  and  that  they  are  applied  appropriately  to  evaluate  the 
contractor  or  MEO. 

This  research  collected  PWS  and  QASP  documents  from  eight  Civil  Engineer 
Operations  Flights  across  the  Air  Force  that  have  completed  or  are  undergoing 
competitive  sourcing.  161  performance  standards  and  metrics  were  identified  and  an 
evaluation  was  conducted  on  them  to  determine  if  the  standards  and  metrics  were 
sufficient  to  evaluate  the  contractor  or  MEO.  The  two-part  evaluation  system  was 
developed  from  metric  design  literature  and  features  from  both  Total  Quality 
Management  and  the  Government  Performance  Results  Act.  The  evaluation  system  was 
also  applied  to  Air  Force  Civil  Engineer  Support  Agency  metrics  and  templates. 

The  results  indicate  critical  areas  of  Civil  Engineer  Operations  Flight  are  not 
sufficiently  evaluated  due  to  insufficient  and  improperly  designed  standards  and  metrics. 
As  a  result  of  this  research,  19  metrics  were  developed  for  evaluating  the  Operations 
Flight  along  with  an  evaluation  system  that  can  be  used  to  assess  the  design  of  metrics 
currently  used  by  any  organization. 
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EVALUATION  OF  CIVIL  ENGINEERING  OPERATIONS  FLIGHT  COMPETITIVE 


SOURCING  STANDARDS  AND  METRICS 


1.  INTRODUCTION 


Background 

The  United  States  Government  seeks  ways  to  inerease  the  effieieney  of 
government  ageneies  and  reduce  costs  of  conducting  business.  One  method  of 
increasing  efficiency  is  through  competitive  sourcing.  The  Department  of  Defense’s 
(DoD)  Competitive  Sourcing  began  in  1955  and  continued  when  the  Office  of 
Management  and  Budget  (OMB)  initiated  Circular  A-76  in  1966  (USGAO-2,  1999,  3). 
Circular  A-76  directed  any  government  activity  identified  as  a  commercial  activity  (CA), 
and  not  classified  as  inherently  governmental,  to  be  competed  against  the  private  sector; 
government  in-house  bids  would  compete  against  private  sector  bids  for  the  commercial 
activity.  A  CA  is  simply  defined  as  a  service  that  is  available  in  the  private  sector 
(USDATL,  2000,  13). 

As  stated  in  the  Vice  President’s  Third  Report  of  the  National  Performance 
Review,  Americans  want  ‘to  get  their  money’s  worth’  and  want  a  Government  that  is 
more  businesslike  and  better  managed  (OMB- 1,  1,  2001).  Therefore,  the  goal  of 
competitive  sourcing  is  to  provide  services  at  minimum  costs  while  sustaining  or 
improving  performance  (USDATL,  2000,  13).  Through  competitive  sourcing,  the 
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government  focuses  on  core  mission  competencies  and  service  requirements  (0MB- 1,1, 

2001). 

Competitive  Sourcing  is  expected  to  trim  costs  by  competing  approximately 
230,000  Department  of  Defense  positions  betweenFiscal  Year  1997  and  Fiscal  Year 
2005  (DoD-1,  2001,  1).  The  expected  savings  over  the  same  eight  years  is  1 1.2  billion 
dollars,  which  could  be  reallocated  for  other  defense  priorities  (DoD,  2001,  1).  Also, 
Competitive  Sourcing  promotes  competition,  whic  h  promotes  improved  performance  at 
reduced  costs  (USDATL,  2000,  14).  Therefore,  it  is  vital  in  these  budget- crunched  years 
that  competitive  sourcing  is  properly  initiated  and  evaluated. 

The  critical  component  of  the  competitive  sourcing  program  is  the  cost 
comparison,  which  competes  government  employees  against  the  private  sector 
(USDATL,  2000,  13),  and  the  key  document  for  the  cost  comparison  is  the  Performance 
Work  Statement  (PWS).  The  PWS  is  a  document  that  is  developed  for  all  activities  being 
solicited  for  contract.  The  PWS  defines  what  service  is  being  requested,  the 
measurements  of  performance  (standards  and  metrics),  and  timeframes  required  (OMB-3, 
2001,  13).  The  PWS  should  also  be  performance  oriented  by  specifying  what  outcomes 
and  measures  are  desired  and  not  placing  directions  on  how  to  achieve  the  outcomes  and 
measures  (OMB-3,  2001,  13).  Ideally,  effective  standards  found  within  the  PWS  will 
lead  to  the  proper  evaluation  metrics  used  to  evaluate  the  performance  and  outputs  of  the 
CA. 

The  standards  found  within  the  PWS  are  the  basis  for  the  formulation  of  metrics 
used  to  measure  the  performance  of  the  service  provider.  The  metrics  used  to  measure 
the  performance  of  the  service  provider  are  found  within  the  Quality  Assurance 
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Surveillance  Plan  (QASP).  The  QASP  is  written  to  cover  the  life  of  the  service  contract 
and  contains  methods  of  surveillance,  the  performance  metrics,  and  sometimes, 
incentives  tied  to  the  performance  of  the  service  provider. 

There  are  two  broad  categories  of  metrics  used  to  evaluate  the  activity  under 
contract:  financial  and  non- financial.  Financial  metrics  gauge  if  the  efforts  to 
competitively  source  have  reduced  overall  activity  costs  and  address  the  question  of  how 
much  money,  if  any,  was  saved.  However,  financial  metrics  “do  not  directly  address  the 
dimensions  of  productivity,  quality,  timeliness,  and  responsiveness  to  customer  needs” 
(Buchheim,  2000,  309).  Therefore,  non-financial  metrics  have  been  established.  Non- 
financial  metrics  evaluate  customer  satisfaction,  quality,  and  productivity.  Together, 
financial  and  non-  financial  metrics  form  the  performance  metrics  used  to  evaluate  the 
service  provider.  The  performance  metrics  are  “a  common  and  mutually  reinforcing 
focus  on  achieving  program  results  and  customer  satisfaction,  measuring  performance, 
and  using  performance  data  to  identify  and  select  improvement  opportunities”  (USGAO- 
4,  1999,  4).  If  the  performance  standards  in  the  PWS  are  written  well,  the  standards  will 
lead  to  a  method  to  evaluate  the  service  provider  to  ensure  the  Government  is  receiving 
the  best  value  for  their  money  while  meeting  mission  requirements. 

This  research  effort  will  focus  on  the  performance  standards  and  metrics  found 
within  the  PWS  and  QASP  documents  of  competitive  sourcing  efforts.  Air  Force 
performance  standards  and  metrics  have  been  evolving  for  at  least  20  years  and  their 
effectiveness  is  not  known.  By  evaluating  performance  standards  and  metrics  of  similar 
competitive  sourcing  efforts,  the  effectiveness  of  the  performance  measurement  can  be 
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evaluated.  The  properly  designed  standards  and  metrics  that  are  found  should  be 
effective  in  supporting  the  Air  Force  mission. 

Problem  Statement 

Performance  standards  and  metrics  currently  used  in  the  competitive  sourcing 
process  may  not  be  sufficient  to  evaluate  performance  in  all  of  the  required  areas  for  the 
service  provider  to  meet  Air  Force  goals. 

Research  Objectives 

•  Evaluate  current  PWS  performance  standards  and  QASP  metrics  for 
proper  design  to  meet  Air  Force  goals 

■  Develop  a  performance  standard  and  metric  evaluation  system 
from  metric  design  literature,  features  from  Total  Quality 
Management  and  Government  Performance  and  Results  Act,  and 
current  Civil  Engineering  metrics 

■  Identify  improperly  designed  standards  and  metrics  and  provide 
recommendations  for  improvement 

•  Identify  Operations  Flight  functional  areas  not  evaluated  by  PWS 
performance  standards  and  QASP  metrics 

Research  Methodology 

This  research  will  be  accomplished  by  a  qualitative  review  of  the  performance 
standards  and  metrics  from  a  representative  sample  of  Air  Force  PWS  and  QASP 
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documents.  The  standards  and  metrics  examined  will  come  from  bases  having  completed 
the  competitive  sourcing  process  and  also  from  bases  currently  undergoing  the  process. 
Performance  standards  and  metrics  will  be  extrapolated  from  the  documents  and 
compiled  for  evaluation.  An  evaluation  system  will  be  developed  from  metric  design 
literature,  components  of  Total  Quality  Management  and  the  Government  Performance 
and  Results  Act,  and  from  additional  inputs  from  Civil  Engineering  metrics.  Both 
qualitative  and  quantitative  evaluation  measures  will  be  used  to  determine  if  the  PWS 
standards  and  QASP  metrics  are  effective  in  supporting  the  Air  Force  mission.  Analysis 
will  focus  on  similarities  and  differences  within  the  measurements  of  performance,  what 
critical  areas  of  the  Civil  Engineering  Operations  Flight  are  being  evaluated  by 
measurements  of  performance,  and  what  areas  are  not  evaluated  by  measurements  of 
performance.  Potential  differences  within  the  measurements  of  performance  will  be 
highlighted  along  with  the  areas  not  evaluated  by  the  measurements  of  performance 
within  the  PWS  standards  and  QASP  metrics. 

Scope  of  Research 

Civil  Engineering  competitive  sourcing  documents  were  sought  from  ten  Air 
Force  Bases  from  around  the  United  States.  The  bases  were  selected  based  upon  criteria 
listed  in  the  methodology.  The  selection  criteria  were  used  to  obtain  a  variety  of  bases, 
therefore,  a  variety  of  competitive  sourcing  documents.  From  within  the  Civil 
Engineering  competitive  sourcing  documents,  only  the  Operations  Flight  standards  and 
metrics  were  chosen  for  evaluation  due  to  the  large  number  of  standards  and  metrics  and 
the  time  commitment  for  evaluation. 
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Relevance 


This  research  will  evaluate  performance  standards  and  metrics  used  in  the 
competitive  sourcing  process.  The  Air  Force  continues  to  compete  commercial  activities 
with  the  private  sector,  and  it  is  imperative  the  Air  Force  spends  taxpayers’  dollars 
effectively  and  efficiently.  The  Air  Force  is  trying  to  obtain  the  best  value  for  its  money 
and  will  continue  to  look  for  ways  to  improve  its  efforts.  By  focusing  on  the 
performance  standards  and  metrics  and  the  outcomes  of  an  activity,  the  Air  Force  will 
have  the  ability  to  measure  the  service  being  provided  and  have  the  ability  to  evaluate  the 
service  to  ensure  the  service  meets  the  needs  of  the  Air  Force.  If  the  standards  and 
metrics  are  well  written,  the  service  provider  has  data  to  guide  improvements.  This  in 
turn  improws  efficiency  and  provides  better  service  for  the  money.  The  results  of  this 
research  will  highlight  areas  of  differences  and  recommend  effective  and  efficient 
standards  and  metrics  to  be  used  in  future  PWS  and  QASP  documents. 

Summary 

This  chapter  provided  background  information  on  the  competitive  sourcing 
process,  the  PWS  document  and  standards,  and  the  importance  of  performance  metrics  in 
the  QASP.  The  research  objectives  were  outlined  as  was  the  methodology  that  was  used 
to  collect  the  information.  Finally,  this  chapter  outlined  the  scope  of  the  research  and  the 
relevance  to  the  Air  Force  mission. 

Chapter  2  provides  the  literature  review  used  in  this  research  effort.  It  discusses 
the  concept  of  outsourcing  and  how  it  relates  to  the  Government.  Chapter  2  provides 
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more  in  depth  information  on  both  the  competitive  sourcing  and  cost  comparison 
processes  and  how  the  Government  conducts  them.  A  detailed  description  is  then  given 
for  the  critical  documents  of  competitive  sourcing,  emphasizing  their  importance  to  the 
competitive  sourcing  process.  Finally,  literature  is  introduced  for  the  proper  design  of  a 
performance  measurement  system.  This  information  provides  an  outline  used  to 
establish  a  performance  standard  and  metric  evaluation  system. 

Chapter  3  discusses  the  competitive  sourcing  process  specific  to  the  Air  Force  and 
the  civil  engineering  community.  Chapter  3  outlines  the  methodology  used  in  selecting 
the  Air  Force  bases  for  this  research  effort.  The  bases  are  identified  and  the  competitive 
sourcing  documents  solicited.  Chapter  3  then  identifies  the  steps  used  in  extracting  and 
compiling  the  performance  standards  and  metrics.  Finally,  the  steps  to  create  the 
evaluation  system  are  discussed  and  then  the  evaluation  system  is  applied  to  the 
performance  standards  and  metrics. 

Chapter  4  contains  the  analysis  of  the  performance  standards  and  metrics.  Each 
Operations  Flight  performance  measurement  is  evaluated  with  the  evaluation  system 
contained  in  Chapter  3.  Each  performance  measurement  evaluation  lists  justifications  as 
to  why  the  measurement  is  or  is  not  properly  designed.  Also,  Civil  Engineer  Operations 
Flight  critical  areas  not  evaluated  by  the  current  standards  and  metrics  will  be  identified. 

Air  Force  Civil  Engineer  Support  Agency  Operations  Flight  metrics  and  templates 
were  also  evaluated  to  identify  critical  areas  not  sufficiently  evaluated.  Using  the  results 
of  the  evaluations,  19  metrics  were  designed  to  adequately  evaluate  all  critical  areas  of 
the  Operations  Flight  as  identified  by  this  research. 
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Chapter  5  documents  the  results  found  by  the  evaluation  of  the  standards  and 
metrics  and  provides  recommendations  on  improving  some  of  the  standards  and  metrics 
and  also  identifies  critical  areas  needing  measurements  of  performance. 
Recommendations  are  also  provided  to  implement  the  designed  metric  evaluation  system 
as  well  as  the  19  proposed  metrics.  Research  limitations  and  future  research  topics  are 
also  introduced. 
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2.  LITERATURE  REVIEW 


The  purpose  of  this  chapter  is  to  provide  background  information  on  the 
importance  and  use  of  performance  standards  and  metrics  and  discuss  potential  problems 
that  exist  today  with  their  use.  The  literature  summary  covers  the  topics  of  outsourcing 
and  the  A-76  process  and  how  they  relate  to  one  another.  A  history  of  the  A-76  process 
is  given  along  with  a  discussion  of  the  cost  comparison  process.  An  introduction  is  then 
given  to  the  critical  documents,  the  Performance  Work  Statements  (PWS)  and  Quality 
Assurance  Surveillance  Plan  (QASP),  used  within  the  A-76  process  followed  by  an 
outline  for  the  design  of  performance  metrics  and  the  general  data  categories  of  metrics 
used  in  this  research.  Finally,  key  features  of  Total  Quality  Management  (TQM)  and  the 
Government  Performance  Results  Act  (GPRA)  will  be  identified  and  their  importance  to 
the  evaluation  of  PWS  performance  standards  and  QASP  performance  metrics  will  be 
discussed. 

Outsourcing 

Outsourcing  is  a  strategic  term  used  by  private  sector  companies  seeking  to 
increase  their  company’s  skill  and  resources.  Outsourcing  can  simply  be  defined  as 
transferring  the  ownership  of  an  internal  process  or  function  to  an  external  supplier 
(Outsourcing  Center,  2001,  2).  By  outsourcing  certain  functions,  companies  can  focus  on 
their  core  competencies  where  they  obtain  “definable  preeminence”  and  can  offer  their 
customers  unique  value  (Quinn,  1994,  43).  Results  of  outsourcing  can  be 
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significant  because  the  outsourcing  company  gains  the  utilization  of  an  external 
company’s  resources,  to  include  investments,  innovations,  and  specialized  capabilities, 
all  of  which  would  be  too  expensive  or  impossible  to  duplicate  internally  (Quinn,  1994, 
43).  It  has  also  been  found  that  outsourcing  can  lower  investments  and  create  better 
responsiveness  to  customer  needs  (Quinn,  1994,  43).  The  findings  suggest  a  company 
can  improve  efficiency  through  the  use  of  an  external  company’s  capabilities,  such  as  its 
innovations  and  specialized  capabilities,  and  improve  efficiency  by  potentially  improved 
customer  responsiveness. 

With  this  in  mind,  research  conducted  on  an  outsourced  public-sector  function 
evaluated  if,  indeed,  there  were  the  savings  and  increased  efficiency.  The  results  were 
less  than  flattering  and  did  not  yield  solid  support  for  outsourcing.  In  the  research,  an 
analysis  was  conducted  to  determine  if  quality  and  efficiency  increased  as  cost  decreased 
by  outsourcing  foster  care  at  group  homes.  From  this  analysis,  significant  stumbling 
blocks  were  discovered.  They  included:  a)  the  efficiency  and  effectiveness  of  processes 
were  difficult  to  define,  b)  cost  data  was  not  maintained  on  in-  house  functions  to  make 
the  post  outsourcing  comparisons,  and  c)  many  contracts  were  not  monitored  efficiently 
(Prager,  1996,  195-200).  The  actual  study  had  to  be  abandoned  due  to  the  lack  of  pre¬ 
outsourcing  financial  data.  The  author  of  the  study  warns  against  getting  too  involved 
in  outsourcing  without  considering  long  term  effects  (e.g.,  higher  hidden  costs  regardless 
of  initial  savings).  Despite  the  lack  of  support  favoring  outsourcing,  many  companies 
still  seek  to  improve  efficiency  and  decrease  cost  by  outsourcing  certain  functions  from 
within  their  organization. 
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The  concept  of  outsourcing  is  not  specific  to  private  sector  companies;  the 
Government  is  also  seeking  ways  to  improve  efficiency.  The  goals  of  competitive 
sourcing  within  the  Government,  specifically  the  Air  Force,  are  much  the  same  as  the 
goals  of  outsourcing.  The  four  goals  of  the  Air  Force  Outsourcing  and  Privatization 
program  are:  1)  sustain  readiness;  2)  improve  efficiency  and  reduce  costs;  3)  create  funds 
for  force  modernization;  4)  focus  on  core  Air  Force  missions  (SAF-1,  1997,  1).  Due  to 
some  controversy  over  the  term  “outsourcing”,  the  A-76  initiatives  became  known  as 
“competitive  sourcing”  within  the  Government. 

One  aspect  of  competitive  sourcing  is  improved  efficiency.  The  improved 
efficiency  of  a  public  organization  can  lead  to  productivity  improvement.  Productivity 
improvement  is  defined  as  the  “production  of  more  and/or  better  services  for  each  tax 
dollar  spent  and  staff  hour  invested”  and  is  a  function  of  the  following  factors:  top 
management  support,  committed  people  at  all  levels,  performance  measurement  system, 
employee  training,  reward  structures,  community  involvement,  and  feedback  and 
correction  mechanisms  (Lee,  2000,  423).  The  factor  most  significant  to  this  research 
effort  is  the  performance  measurement  system. 

The  performance  measurement  system  is  a  key  component  of  “managing  for 
results”,  which  is  an  old  ideal  in  Government  (Aristigueta,  2001,  254).  The  government 
has  continually  sought  ways  to  manage  for  results.  Recently,  the  shift  has  taken  the 
Government  from  focusing  on  inputs  and  processes  to  focusing  on  outcomes  or  results 
(Aristigueta,  2001,  254).  By  focusing  on  the  outcomes  of  the  processes,  the  “managing 
for  results”  theory  is  that  effectiveness,  efficiency,  and  accountability  of  Government  will 
improve,  which  will  lead  to  reduction  in  costs,  increase  in  public  satisfaction  with  the 
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Government,  and  impro\ement  in  quality  of  services  (Aristigueta,  2001,  254).  Managing 
for  results  is  vital  to  reaching  the  goals  established  by  the  mission  of  the  Air  Force. 

By  identifying  and  selecting  functions  to  undergo  competitive  sourcing,  the 
Government  seeks  to  improve  efficiency.  This  efficiency  improvement  can  only  be 
evaluated  if  a  properly  designed  performance  measurement  system  has  been 
implemented.  Prior  to  the  introduction  of  a  performance  measurement  system,  the 
history  of  Government  competitive  sourcing  is  outlined  in  the  following  section. 

Competitive  Sourcing  lA-76) 

Current  Office  of  Management  and  Budget  Circular  (OMBC)  A-76  policy  was 
established  through  many  revisions  of  the  Bureau  of  Budget  Bulletin  55-4.  Bulletin  55-4 
was  published  in  1955  with  the  basic  premise  that  the  Government  should  not  be  in  the 
business  of  competing  with  its  citizens.  This  bulletin  created  a  policy  in  which  the 
Government  relied  upon  the  services  of  the  local  community  without  the  cost  comparison 
concept.  As  a  result,  the  services  sought  after  were  not  cost  effective  and  therefore,  they 
were  not  in  the  best  interest  of  the  taxpayers  (OMB-4,  2001,  3). 

In  1957,  the  bulletin  was  updated  to  include  the  basic  concepts  of  cost  comparison 
to  direct  the  government  in  obtaining  the  best  service  value.  The  Bureau  of  Budget  was 
transformed  into  the  Office  of  Management  and  Budget  (0MB),  and  in  1966,  0MB 
issued  Circular  A-76  that  stated  the  cost  comparison  guideline  found  within  the  Circular 
supported  the  Government’s  efforts  to  continue  to  rely  on  the  private  sector  (0MB- 1, 
2001,  1-2).  OMBC  A-76  was  revised  several  times  (e.g.,  1979,  1983,  1986,  and  in 
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1999),  giving  guidance  on  conducting  a  cost  comparison  and  further  refining  the  cost 
comparison  concept. 

The  1999  revision  to  OMBC  A-76  (previously  listed)  defines  the  competitive 
sourcing  process  used  today.  This  revision  contains  three  initiatives  vital  to  the 
competitive  sourcing  process:  achieve  economy  and  enhance  productivity  (through 
competition),  retain  governmental  functions  in-house  (certain  functions  are  inherently 
governmental),  and  rely  on  the  commercial  sector  (competitively  source  functions 
identified  as  a  commercial  activity)  (0MB- 1,  2001,  2).  The  “competition”  and  “rely  on 
commercial  sector”  initiatives  are  the  focus  of  the  cost  comparison  discussed  in  the  Cost 
Comparison  Process  section. 

OMBC  A-76,  along  with  Air  Force  Instruction  38-203  and  numerous  other 
Department  of  Defense  guidebooks  (e.g.,  Cost  Comparison  Handbooks  and  Commanders 
Handbook  on  Competitive  Sourcing),  define  the  structured  process  of  the  cost 
comparison  in  which  the  Government  competes  against  the  private  sector  for  the  services 
desired.  This  competition,  resulting  from  the  cost  comparison,  allows  commanders  to 
make  better  business  decisions  regardless  of  the  service  provider  (USDATL,  2000,  14). 
The  current  Circular  attempts  to  maintain  a  balance  of  interests  between  federal 
managers,  employees,  and  the  private  sector  with  that  of  federal  taxpayers  (0MB- 1, 
2001,  1).  The  steps  of  the  A-76  process  leading  to  the  cost  comparison  and  eventual 
service  provider  are  found  in  Figure  1,  Outline  of  A-76  Process. 
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Step? 

Implement  winner  of 
Cost  Comparison 


Figure  1.  Outline  of  A-76  Process 

(Adapted  from  USDATL,  2000,  17) 


Step  1  of  Figure  1  involves  the  classification  of  the  function  as  either  inherently 
governmental  or  as  a  commercial  activity  by  the  Headquarters  of  the  United  States  Air 
Force  (HQ  USAF).  Functions  classified  as  inherently  governmental  are  retained  in-house 
and  those  classified  as  a  commercial  activity  can  be  competitively  sourced  (OMB-3, 

2001,  2).  Step  2  is  the  public  announcement  of  the  function  to  be  competitively  sourced. 
This  announcement  begins  the  clock  for  the  time  allowed  to  complete  a  study;  24  months 
from  the  announcement  to  implementation  for  a  single  function  and  48  months  from  the 
announcement  to  implementation  for  multiple  functions  (ACC,  2001,  4).  Step  3  involves 
the  development  of  the  Performance  Work  Statement  (PWS)  and  Quality  Assurance 
Surveillance  Plan  (QASP).  The  PWS  lists  the  requirements  the  government  is  seeking 
for  the  function  and  the  QASP  is  the  document  that  outlines  the  Government’s  contract 
inspection  plan.  Steps  4a  and  4b  are  conducted  simultaneously  once  the  PWS  has  been 
released.  The  Government,  through  contracting  personnel  and  subject  matter  experts, 
establishes  its  bid  for  the  commercial  activity  based  upon  the  information  found  in  the 
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PWS  while  and  the  private  sector  companies  develop  their  individual  bids  for  the 
commercial  activity  based  upon  the  infonnation  found  in  the  PWS.  The  Government  bid 
becomes  known  as  the  Most  Efficient  Organization  (MEO).  The  MEO  is  defined  as  the 
Government’s  estimate  of  the  resources  required  to  perform  the  commercial  activity  and 
can  be  a  mix  of  civilian  employees,  military  members  in  special  cases,  and  other  contract 
support  necessary  to  complete  the  Commercial  Activity  (AFMIA-1,  n.  pag.).  Step  5  is 
managed  by  contracting  and  involves  the  evaluation  of  the  private  sector  bids  and 
choosing  one  winner  among  them.  Step  6  is  the  cost  comparison  that  competes  the  MEO 
bid  against  the  winner  of  the  private  sector  bids.  The  winner  of  the  cost  comparison  is 
chosen  based  upon  “best  value”  (discussed  in  the  Best  Value  Approach  section)  to  the 
Government  and  then,  barring  any  protests  or  appeals,  contract  implementation.  Step  7,  is 
initiated. 

The  PWS/QASP  documents  are  pivotal  within  the  A-76  process  and  will  be 
further  discussed  in  the  PWS  and  QASP  section.  Quality  MEO  and  Private  Sector  bids 
rely  upon  these  documents  to  lead  to  a  fair  and  reasonable  cost  comparison  and  a 
successful  contract  implementation  for  the  service  provider. 

Best  Value  Approach 

The  goal  of  every  acquisition  is  to  provide  the  Government  the  best  value 
regardless  of  the  acquisition  process  used.  All  competitive  methods  listed  in  the  Federal 
Acquisitions  Regulations  (FAR)  are  acceptable  to  use  in  the  A-76  process  (i.e.,  sealed 
bid,  two-step,  and  other  competitive  negotiated  procurement  techniques)  (DAF-1,  2000, 
86).  The  best  value  concept  compares  factors  such  as  past  performance  and  cost  of  the 
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private  sector  proposal  and  makes  trade-offs  (e.g.,  pay  more  if  service  provider  has 
proven  past  performance)  to  ensure  the  Government  is  obtaining  the  better  value 
(USGAO-2,  1999,  10).  The  objective  of  A-76  and  the  best  value  concept  is  to  obtain  the 
greatest  benefit  for  the  Government  regardless  of  who  provides  the  service.  Through  the 
process  outlined  in  Circular  A-76,  the  Air  Force  attempts  to  obtain  best  value  by 
promoting  competition  to  improve  performance  and  reduce  cost.  To  ensure  sustained  or 
improved  performance  is  being  achieved  and  Federal  taxpayers  are  getting  the  best  value 
for  their  money,  performance  metrics  must  be  introduced  and  tracked  prior  to  the  cost 
comparison  process. 

Cost  Comparison  Process 

At  the  heart  of  competitive  sourcing  is  the  cost  comparison  process.  The  cost 
comparison  process  focuses  on  two  bids:  the  MEO  and  the  winner  of  the  private  sector 
bids  (OMB-3,  2001,  3),  found  in  Step  5  of  Figure  1,  Outline  of  A-76  Process.  Once  the 
private  sector  bids  have  been  reviewed  and  the  bid  that  represents  the  best  value  from  the 
private  sector  has  been  selected  (Step  5  of  Figure  1),  the  private  sector  bid  and  the  MEO 
bid  are  compared  and,  again,  the  winner  of  the  two  bids  is  selected  based  upon  the  best 
value  for  the  Government. 

The  cost  comparison  process  contains  two  critical  documents  that  determine  a 
successful  cost  comparison:  the  Performance  Work  Statement  (PWS)  and  the  Quality 
Assurance  Surveillance  Plan  (QASP)  (Step  3,  Figure  1).  Both  documents  contain  critical 
performance  measurements  that  are  vital  in  evaluating  the  performance  of  the  service 
under  contract.  Without  the  performance  measurements,  the  quality  of  the  service  cannot 
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be  evaluated  to  determine  if  the  Government  and  federal  taxpayers  are  getting  the  best 
value. 

PWS  and  OASP 

The  following  sub  -  headings,  Pez/ormawce  Work  Statements  (PWS)  and  Quality 
Assurance  Surveillance  Plan  (QASP),  outline  the  PWS  and  QASP  documents, 
respectively,  and  provide  background  infomiation  on  their  significance  to  performance 
standards  and  metrics. 

Performance  Work  Statement  (PWS) 

The  PWS,  as  part  of  the  solicitation,  is  the  most  critical  document  of  the  A-76 
cost  comparison  because  all  documents  submitted  by  the  Government  and  the  private 
sector  regarding  the  solicitation  are  based  upon  the  PWS  (USDATL,  2000,  22).  All 
remaining  documents  within  the  A-76  process  rely  on  what  is  written  in  the  PWS  (to 
include  the  Private  Sector  bid,  MEO,  and  transition  plans,  used  to  complete  the  transfer 
of  the  function  once  the  service  provider  is  determined).  Therefore,  it  is  imperative  the 
information  contained  within  the  PWS  is  accurate  because  a  successful  A-76  program 
relies  on  the  quality  of  the  PWS.  The  PWS  includes  infomiation  on  the  requirements, 
performance  measures  and  standards,  workload,  and  conditions  of  performance  for  the 
service  being  reviewed  (OMB-3,  2001,  13).  The  PWS  is  written  to  emphasize  results 
rather  than  dictating  how  a  service  provider  must  perform  the  service;  a  PWS  should  be 
performance- oriented.  Some  key  areas  resulting  from  a  quality  PWS  should  be: 
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maximized  performance,  maximized  competition  and  innovation,  maximized 
accountability  by  allowing  service  provider  to  be  responsible  fcr  process,  reduced  risk, 
increased  efficiency  and  effectiveness,  simplified  contract  administration  through 
implemented  performance  measurement;  all  of  which  can  be  accomplished  through  the 
collection  of  data,  analysis  of  data,  and  documentation  of  appropriate  data  (USDATL, 
2000,  22).  To  determine  if  any  of  the  key  areas  are  improving,  data  must  be  collected 
prior  to  the  transfer  to  the  service  provider  and  must  be  continually  tracked  after  the 
service  provider  assumes  control  of  the  function. 

Collecting  and  analyzing  data  are  important  components  when  designing  the 
requirements  for  the  PWS  because  the  correct  data  must  be  collected  and  properly 
analyzed  to  offer  significant  information  to  provide  and  potentially  improve  the  service  of 
the  function  that  is  being  considered  for  contract.  Also,  data  improperly  documented  will 
not  provide  accurate  information  on  the  improvement  in  results.  Therefore,  it  is  vital  to 
take  time  to  plan  for  the  development  of  the  PWS. 

Developing  a  PWS 

The  four  steps  taken  to  develop  the  PWS  can  be  seen  in  Figure  2  and  each  step 
will  be  outlined  in  the  following  four  sub -sub- headings. 
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Figure  2.  PWS  Development 

(adapted  from  USDATL,  2000,  26) 


Team  Formation 


Step  1  is  the  formation  of  a  team  to  develop  the  PWS.  When  ereating  a  PWS,  a 
team  manager  should  choose  the  most  qualified  and  experienced  personnel  to  be  on  the 
team  (USDATL,  2000,  26).  The  team  should  be  comprised  of  personnel  from  the 
functional  office,  the  contracting  office,  and  the  management  analysis  office  (USDATL, 
2000,  27).  Positions  within  the  PWS  team  include  a  team  leader,  subject  matter  experts, 
contracting  officer,  management  analysts,  and  editors.  The  PWS  team  leader  has 
authority  over  developing  the  PWS.  The  subject  matter  experts  are  the  functional  area 
experts  that  identify  the  service  required  of  the  service  provider  and  make 
recommendations  on  the  acceptance  and  measurement  of  the  service  provided  (USDATL, 
2000,  27).  These  subject  matter  experts  are  key  personnel  in  determining  the 
performance  requirements  of  the  contract.  One  of  the  duties  provided  by  the  Contracting 
Officer  while  assigned  as  a  PWS  team  member  is  to  ensure  the  Request  for  Proposal 
(developed  from  the  PWS)  meets  all  the  requirements  of  a  contract  document  as  stated  in 
the  Federal  Acquisition  Regulation  (FAR).  The  management  analysts  perform  additional 
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data  collection  and  typically  are  consultants  or  in-house  resources  (USDATL,  2000,  27). 
Finally,  the  editors  help  to  edit  and  format  the  final  document. 


Plan  of  Action  and  Milestone  (POAM) 


Step  2  of  the  PWS  planning  is  the  development  of  the  Plan  of  Action  and 
Milestone  (POAM).  The  POAM  summarizes  the  actions  to  be  taken  in  developing  the 
PWS  and  establishes  milestones  for  completion  of  the  actions  (USDATL,  2000,  29).  The 
POAM  outlines  the  steps  to  a  typical  PWS  development  up  to  150  days.  Within  the  150 
days.  Day  22  through  Day  73  involve  the  development  of  performance  outcomes  and 
measures  and  an  example  can  be  seen  in  Table  1,  Determination  of  performanee 
outcomes  and  measures.  The  entire  150  day  POAM  can  be  found  in  Appendix  A.  Day 
22  through  Day  73  are  critical  to  the  development  of  the  PWS  and  account  for  34%  of  the 
days  to  develop  a  PWS.  The  two  categories  listed  emboldened  in  Table  \ ,  Identifying 
Activity  Goals  and  Developing  Performance  Measures,  are  the  two  major  categories  for 
performance  measurement,  and  the  remaining  steps  are  subsets  of  the  major  categories. 


Table  1.  Determination  of  Performance  Outcomes  and  Measures 


Action 

Day 

Identifying  Activity  Goals 

22-42 

Just-In  Time  Training  on  this  Action 

22-23 

Initial  Data  Collection  and  Analysis 

22-29 

Data  to  Identify  Activity  Goals 

22-29 

Other  General  Data  to  Write  the  PWS 

22-29 

Initial  Data  Analysis 

22-31 

Identification  of  the  Purpose 

29-36 

Identification  of  the  Goals 

29-42 

Developing  Performance  Outcomes 

29-56 

Just-In  Time  Training  on  this  Action 

29-30 

Identify  Additional  Data  Collection  and  Analysis 

29-36 

Performance  Outcome  Analysis 

29-56 

Identification  of  Outcome  Owner 

36-56 

Verification  of  Purposes,  Goals  and  Services 

45-56 
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Developing  Performance  Measures 

36-73 

Just-ln  Time  Training  on  this  Action 

36-37 

Identify  Additional  Data  Collection  and  Analysis 

36-43 

Select  Performance  Measures 

36-73 

Document  Supporting  Data 

43-73 

Identify  Performance  Measure  Owner 

50-73 

Document  Sources  and  Location  of  Data 

50-73 

Document  Methodology  for  Completion  and  Management  of  Data 

50  -73 

Identify  Source  of  Performance  Measure 

50-73 

Deyelop  Performance  Standards 

36-73 

Quality  Standard 

36-73 

Timeliness  Standard 

36-73 

Quantity  Standard  (Workload) 

36-73 

Establish  Baseline 

50-73 

Verify  Purposes,  Goals  and  Outcomes 

50-73 

(USDATL,  2000,  30) 


The  51  days  projected  for  developing  performance  outcomes  and  measures 
indicate  that  performance  measurements  are  significant  in  the  cost  comparison  process 
and  an  effort  must  be  made  to  properly  develop  them. 


PWS  Decision  Plan 


The  PWS  Decision  Plan  is  a  tool  used  to  identify  and  mitigate  any  potential 
problems  that  may  exist  in  the  development  of  the  PWS  (USDATL,  2000,  31). 

Handbook  #4  outlines  the  decision  plan  and  includes  six  steps  to  help  in  the  development 
of  the  PWS.  The  first  step  of  the  PWS  decision  plan  is  determining  the  scope  of  the  Cost 
Comparison  prior  to  developing  the  PWS.  The  scope  estimates  the  required  work  to  be 
performed  by  the  service  provider  (USDATL,  2000,  32).  Step  two  of  the  PWS  decision 
plan  is  the  forecasting  of  the  workload.  Step  three  of  the  PWS  decision  plan  addresses 
the  availability  of  performance  measures  and  standards,  and  historical  workload  data.  If 
performance  measures  and  standards  are  not  available  and  historical  data  are  non- 


21 


existent,  more  effo  rt  and  time  will  be  required  to  develop  the  performance  measures  and 
standards  (USDATL,  2000,  32).  Step  four  of  the  PWS  decision  plan  identifies  the  length 
of  the  contract,  typically  a  one-year  contract  with  four  optional  years  (USDATL,  2000, 
33).  Step  five  of  the  PWS  decision  plan  involves  deviations  and  waivers  from  current 
directives.  This  step  is  related  to  process  improvement  and  innovation  practices  sought 
after  by  the  Cost  Comparison  process  (USDATL,  2000,  33).  Finally,  step  six  of  the  PWS 
decision  plan  is  identifying  potential  surveillance  methods  (USDATL,  2000,  33).  The 
identified  surveillance  methods  are  critical  in  evaluating  if  the  Government  is  obtaining 
sustained  or  improved  performance  from  the  service  provider. 

Plannins  Data  Collection  and  Analysis 

The  final  step  in  the  PWS  development  is  the  planning  for  the  collection  of  data. 

It  is  important  to  think  of  what  data  will  be  required  in  the  PWS  for  evaluation  purposes, 
how  the  data  will  be  used,  and  who  will  provide  the  data  most  efficiently  (USDATL, 
2000,  34).  The  data  typically  required  of  a  PWS  focuses  around  the  historical  workload 
of  the  function.  The  historical  workload  data  will  identify  function  activities,  purposes, 
and  goals.  For  example,  a  facilities  maintenance  function  relies  heavily  on  a  maintenance 
file  system  that  identifies  work  requirements  and  where  resources  have  been  focused.  If 
the  facility  maintenance  function  were  to  be  competitively  sourced,  data  would  need  to  be 
obtained  to  identify  the  amount  of  work  expected  (to  be  written  in  the  solicitation),  and 
what  kind  of  work  is  accomplished.  The  facility  maintenance  personnel  would  look  to 
obtain  the  historical  data  from  their  maintenance  file  system.  When  the  historical  data 
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are  interpreted  properly,  a  quality  PWS  can  be  written  which  leads  to  improved 
performance  (outcomes)  of  the  commercial  activity. 

PWS  Format 

To  promote  quality  and  consistent  PWS  documents,  Air  Force  Instruction  (AFI) 
63-124  establishes  guidelines  for  the  format  of  the  PWS.  The  PWS  includes:  a 
Description  of  Services,  a  Service  Delivery  Summary  (SDS),  Government-Furnished 
Property  and  Services,  and  General  Information  (SAF-2,  1999,  6).  The  SDS  is  required 
in  every  PWS  and  is  the  critical  section  of  the  PWS  because  it  identifies  the  performance 
objective  (service  required  of  the  service  provider)  and  performance  threshold  (specific 
standard  for  which  the  service  provider  will  be  held  accountable)  of  each  task  within  the 
PWS. 

The  SDS  portion  of  the  PWS  will  be  used  in  this  research  effort  to  identify  the 
various  differences  in  the  performance  measures  and  standards  from  base  to  base.  An 
example  of  a  portion  of  the  information  found  within  a  SDS  can  be  seen  in  Table  2, 
Service  Delivery  Summary  Example,  which  was  extracted  from  a  civil  engineering 
squadron  PWS  template  provided  on  the  Air  Force  Civil  Engineering  Support  Agency 
(AFCESA)  website. 


Table  2.  Service  Delivery  Summary  Exam] 

pie 

Performance  Objective 

Performance  Threshold 

Treat  customers  politely,  cheerfully  and  promptly 

1.1.1 

Customer  service  rating  of 
at  least  4.0  on  a  5.0  scale 

Respond  to  and  complete  emergency,  urgent,  and  routine 
service  calls 

1.1.3. 

95%  of  required  time 
limits 

Maintain,  repair,  construct,  and  operate  the  supporting 
infrastructure  ensuring  cost  effective  and  reliable  support 

1.2. 

100%  of  time 
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Provide  economical  maintenance,  repair,  construction. 

1.3. 

95%  of  scheduled 

installation,  operation,  and  service  functions  for  real  property. 

inspections  and/or  work 

Real  Property  Installed  Equipment  (RPIE),  and  designated 
Equipment  Authorized  Inventory  Data  (EAID) 

completed  on  time 

Table  2  identifies  the  performance  objective  to  be  completed  by  the  service 
provider.  It  identifies  the  “what”  of  a  task.  The  table  also  identifies  the  paragraph  of  the 
PWS  or  SOW  where  the  requirement  can  be  found.  Each  requirement  has  a  specific 
location  within  the  PWS  or  SOW  and  is  typically  identified  by  chapter  numbers,  followed 
by  paragraph  and  sub -paragraph  numbers.  For  example,  Table  2  contains  a  performance 
objective  stating  the  service  provider  must  “treat  customers  politely,  cheerfully,  and 
promptly”,  which  can  be  found  within  sub-paragraph  one,  of  the  first  paragraph  of 
Chapter  one  of  the  particular  SOW.  Finally,  Table  2  establishes  the  threshold  value  to 
which  the  service  provider  will  be  held  accountable.  The  information  contained  in  the 
performance  threshold  should  not  dictate  to  the  service  provider  “how”,  but  establish  the 
threshold  of  performance  that  will  be  measured  (the  results  of  the  task).  The  SDS 
portions  of  the  PWS  documents  provide  the  minimum  performance  required  of  the 
service  provider,  which  are  evaluated  in  this  research  effort. 

Quality  Assurance  Surveillance  Plan  (QASP) 

The  Quality  Assurance  Surveillance  Plan  is  a  vital  document  for  the  oversight  of 
the  performance  for  the  service  provider.  The  QASP  links  the  written  standards  of  the 
PWS  to  the  performance  of  the  provider.  The  document  describes  the  inspection  methods 
that  will  be  used,  the  necessary  reports  written  by  quality  assurance  personnel  and  the 
resources  (i.e.,  number  of  personnel)  and  estimated  work  hours  used  in  the  oversight  of 
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the  contract  (USDATL,  2000,  22).  The  QASP  is  written  to  assist  the  contracting 
officer’s  performance  evaluation  of  the  service  provider  and  defines  the  process  by  which 
the  Government  will  evaluate  the  performance  of  the  service  provider  and  evaluate  the 
compliance  of  the  service  provider  with  PWS  standards  (USDATL,  2000,  23).  The 
document  contains  the  frequency,  purpose,  and  method  of  each  inspection,  along  with  the 
penalties  of  not  meeting  the  performance  standards  listed  within  the  PWS.  The  QASP 
may  or  may  not  accompany  the  PWS  when  the  solicitation  package  is  submitted,  but  it 
must  be  implemented  regardless  of  who  wins  the  Cost  Comparison  (USDATL,  2000,  23). 

The  QASP  is  developed  by  the  Business  Requirements  Advisory  Group  (BRAG), 
which  is  established  by  the  contracting  officer  and  is  a  customer- focused  multifunctional 
team  (SAF-2,  1999,  2).  The  BRAG  is  established  to  plan  and  manage  service  contract 
outcomes  to  the  satisfaction  of  its  customers  throughout  the  life  of  the  requirement  (SAF- 
2,  1999,  2- 10).  The  BRAG  is  not  to  be  confused  with  the  PWS  team  previously 
mentioned  in  the  Team  Formation  section.  The  BRAG  has  an  entirely  different  function, 
but  may  contain  some  of  the  same  personnel  as  the  PWS  team.  Once  the  PWS  team  has 
developed  the  PWS,  the  team  is  disbanded;  the  BRAG  then  imposes  surveillance 
requirements  to  the  SDS  (from  the  PWS),  resulting  in  a  QASP  that  is  identical  to  the 
PWS  SDS  except  for  the  surveillance  requirements.  An  example  of  a  portion  of  a  QASP 
can  be  seen  in  Table  3,  Quality  Assurance  Surveillance  Plan  Example.  The  information 
in  Table  3  was  obtained  from  a  civil  engineering  squadron  QASP  template  also  found  on 
the  AFCESA  website.  It  is  interesting  to  note,  the  template  given  did  not  contain  the 
required  surveillance  information  and  therefore.  Table  3  is  identical  to  Table  2,  Service 
Delivery  Summary  Example. 
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Table  3.  Quality  Assurance  Surveillance  PI 

an  Example 

Performance  Objective 

Performance  Threshold 

Treat  customers  politely,  cheerfully  and  promptly 

1.1.1 

Customer  service  rating  of  at 
least  4.0  on  a  5.0  scale 

Respond  to  and  complete  emergency,  urgent,  and 
routine  service  calls 

1.1.3. 

95%  of  required  time  limits 

Maintain,  repair,  construct,  and  operate  the 
supporting  infrastructure  ensuring  cost  effective  and 
reliable  support 

1.2. 

1 00%  of  time 

Provide  economical  maintenance,  repair, 
construction,  installation,  operation,  and  service 
functions  for  real  property.  Real  Property  Installed 
Equipment  (RPIE),  and  designated  Equipment 
Authorized  Inventoiy  Data  (EAID) 

1.3. 

95%  of  scheduled  inspections 
and/or  work  completed  on  time 

Several  bases  have  varying  formats  of  the  QASP.  QASP  formats  usually  have 
the  objeetive  and  performance  standard  or  threshold  columns  (as  shown  in  Table  3),  but 
two  additional  columns  are  often  added  to  the  QASP  format:  one  column  to  identify  the 
surveillance  method  that  will  be  used  to  oversee  the  service  provider  (as  required  in  API 
63-124)  and  another  column  identifying  incentives  if  the  performance  requirements  are 
met.  Surveillance  methods  can  include  customer  comment  cards,  1 00%  inspections,  or 
random  inspections.  The  incentive  column  has  \arying  percentage  amounts  listed  if  the 
performance  requirement  is  met.  For  example,  if  a  service  provider  met  a  requirement 
for  submitting  reports,  a  maximum  payment  percentage  of  2.0  would  be  awarded  to  the 
service  provider.  Table  4,  Additional  QASP  Format,  identifies  a  portion  of  an  additional 
QASP  format  used.  The  incentives  column  was  added  with  sample  data  to  depict  what  a 
QASP  may  look  like  if  it  contained  an  incentives  column.  The  values  found  in  the 
incentives  may  range  from  one  to  five  percent,  and  not  all  standards  or  metrics  would 
have  a  value  in  the  incentives  column. 
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Table  4.  Additional  QASP  Formats 


Performance  Standard 

Method  of 

Surveillance 

Contractor 
provided 
quality  control 

Maintain  a  quality  control  plan  at  a 
level  that  minimizes  customer 
complaints. 

Periodic  review 
of  customer 
complaint  and 
follow-up 
documentation. 

2% 

Civil 

Engineering 
(J-1,  2.0-  1) 

100%  of  taskings  shall  meet  established 
suspenses  with  no  more  than  5% 
requiring  correction. 

Review 
monthly 
metric(s)  and 
customer 
complaint. 

2% 

Infrastructure 
(J-1,  2.1-  1) 

No  facility/infrastructure  related 
discrepancy  shall  prevent  Alert  aircraft 
ability  to  depart  with  1 5  minute  notice. 

Customer 

complaint. 

2% 

Infrastructure 
(J-1,  2.1  -2) 

Pest  Control  Services:  Develop  and 
comply  with  the  IPM  Plan,  approved 
by  the  government.  Plan  developed, 
submitted,  and  approved  on  time. 

Record  daily  pesticide  usage  and  report 
usage  quarterly  to  ACC.  Reduce  yearly 
pesticide  consumption  50%  in 
compliance  with  1993  DOD  baseline 
study. 

Review 

quarterly 

metrics. 

2% 

(adapted  from  Offliti 

tAFB  QASP,  2001,4) 

One  difference  between  Table  4,  Additional  QASP  Format,  and  Table  3,  Quality 
Assurance  Surveillance  Plan  Example,  is  that  Table  4  contains  the  information  for 
locating  the  performance  objective  within  the  objective  cell.  For  example,  the  objectives 
for  infrastructure  can  be  found  in  section  J-1  of  the  PWS.  The  “2.1”  identifies  the 
specific  paragraph  and  the  “-1”  identifies  the  first  objective  for  infrastructure.  “J-1,  2.1- 
2”  is  the  second  objective  for  infrastructure  found  in  Section  J-1,  paragraph  2.1. 
Together,  the  PWS  and  the  QASP  documents  contain  the  information  used  in  the 
measurement  of  performance  within  the  competitive  sourcing  process. 
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Performance  Measurement 


This  section  will  identify  the  importance  of  a  performance  measurement  program 
and  stress  the  significance  of  metric  design.  Relewit  metric  design  literature  will  outline 
what  a  metric  should  include  to  properly  evaluate  a  function  followed  by  an  introduction 
to  two  performance  measurement  programs  used  by  the  Government  to  evaluate 
performance. 

Creatins  Performance  Metrics 

The  performance  standards  and  performance  measures  that  are  created  form  the 
basis  of  the  measurement  program  used  to  evaluate  the  task.  The  performance  standards 
are  the  objectives  being  sought  by  the  task,  and  the  performance  measures  are  the  metrics 
used  to  measure  the  results  of  the  task.  Managers  use  measurement  programs  to  inform 
their  people  about  the  processes  of  the  organization  (Edberg,  1997,  33).  Key  elements 
within  the  measurement  program  are  specific,  quantifiable  performance  metrics;  metrics 
are  quantitative  values  obtained  by  measuring  certain  characteristics  of  a  process 
(Edberg,  1997,  33).  For  example.  Table  3,  Quality  Assurance  Surveillance  Plan  Example 
contains  the  “respond  to  and  eliminate  service  calls”  standard,  which  is  measured  by  the 
number  of  service  calls  received  and  the  number  of  service  calls  completed  within  the 
specified  time  frames  (typically  found  in  the  PWS).  The  resulting  metric  is  then  used  to 
evaluate  the  process  to  determine  if  process  improvements  can  be  made.  The  response 
and  completion  time  standards  (established  by  the  Emergency,  Urgent,  or  Routine 
classification  found  within  AFI  32-1001)  and  the  metric  to  measure  the  response  (may 
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include  a  record  of  the  number  of  calls,  classification  of  each  call,  time  to  respond  to  each 
call,  and  time  to  complete  work)  create  the  measurement  program  for  that  particular  task. 

Problems  have  arisen  when  developing  a  metric  because  the  metric  may  have,  for 
example,  sought  the  wrong  data  or  the  metric  may  have  been  confusing  or  not 
quantifiable.  An  example  of  a  metric  that  is  not  quantifiable  would  be  if  the  standard 
stated  “Operate,  maintain,  and  repair  the  heating  system”  with  a  threshold  of  100%  of  the 
time.  The  “operate,  maintain,  and  repair”  tries  to  include  too  much  into  the  one  metric 
and  is  very  broad.  It  is  not  understandable  to  operate,  maintain,  and  repair  100%  of  the 
time. 

To  help  eliminate  the  problems  with  initiating  a  measurement  program,  a  nine- 
step  guideline  has  been  established  by  the  Department  of  Energy.  The  steps  of  this 
process  are  depicted  in  Table  5,  Steps  to  Creating  a  Performanee  Measurement  Program. 
The  steps  include  identifying  all  organizations  affected  by  the  task  and  determining  all 
requirements;  establishing  standards,  metrics  and  a  collection  procedure;  and  reviewing 
the  performance  and  identifying  improvement  opportunities. 


Table  5.  Steps  to  Creating  a  Peri 

’ormance  Measurement  Program 

Steps 

Purpose 

•  Involve  all  affected  organizations  in  the 
development  of  performance  metrics 

Ensure  that  all  affected  organizations  will  accept 
the  results  of  the  effort 

•  Flow  chart  the  applicable  process 

•  Determine  what  is  important  to  the 
customer 

Identify  critical  activities  (i.e.  “control  points”)  to 
measure,  and  the  results  which  are  worthy  of  being 
measured 

•  Establish  the  performance  measurements 
(i.e.  unit  of  measure,  sensor,  and 
frequency) 

•  Establish  goals  or  standards 

•  Identify  responsible  parties  for  data 
collection,  analysis,  and  reporting 

Collect  the  data,  and  ensure  that  the  data  collection 
process  functions  properly 

•  Analyze  and  report  the  actual 
performance 

Determine  what  actions  should  be  taken  in 
response  to  a  variance.  It  may  be  appropriate  to: 
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•  Compare  actual  performance  with 

>  Ignore  it  (if  the  variance  is  not 

standard  or  goal 

statistically  significant) 

•  Evaluate  causes  of  variances,  and 

>  Fix  it  (if  it  is  significant,  or  indicates  an 

potential  corrective  actions 

unfavorable  trend) 

>  Challenge  the  goal  (if  achieving  the  goal 

would  be  counter-productive  to  more 

important  Corporate  objectives) 

>  Challenge  the  metric  (if  the  metric  is 

providing  useless  or  hard -to -interpret 

information) 

(Table  5  adapted  from  Buchiem,  2000,  311) 


The  “establish  the  performanee  measurements”  step  of  Table  5,  Steps  to  Creating 
a  Performance  Measurement  Program,  diseusses  the  formation  of  metrics.  The  metric 
will  be  used  to  evaluate  the  function  and  it  must  be  properly  designed  to  assess  the 
function  properly.  A  well  designed  performance  metric  must  include  three  elements:  “1) 
a  defined  measure  of  unit,  2)  a  ‘sensor’  which  gathers  and  records  the  raw  data,  and  3)  a 
frequency  with  which  measurements  and  reports  are  to  be  made”  (Buchheim,  2000,  310). 
The  response  time  metric  in  Table  3,  Quality  Assurance  Surveillance  Plan  Example  has 
time  as  the  defined  measure  of  unit.  The  sensor  is  the  system  that  collects  the 
information,  typically  the  Workgroup  Information  Management  System  (WIMS),  an  Air 
Force  data  system.  The  frequency  is  determined  at  the  unit  level  based  upon  the 
importance  of  the  work  being  evaluated.  The  metric  reporting  could  be  on  a  monthly 
basis  or  a  quarterly  basis,  but  in  either  case,  each  work  order  response  time  must  be 
tracked  within  the  specified  period. 

Along  with  the  three  elements  previously  mentioned  (a  defined  measure  of  unit,  a 
sensor,  and  a  frequency),  the  performance  metric  must  also  be:  1)  understandable;  not 
difficult  to  define  or  understand,  2)  quantifiable;  objective  with  much  of  the  personal 
influence  or  judgment  reduced,  3)  cost-effective;  value  of  information  sought  must 
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exceed  the  cost  of  data  collection,  4)  proven;  validated  to  have  shown  a  drive  to 
improvement,  and  5)  high  impact;  collection  of  metrics  must  be  worthwhile  (Edberg, 
1997,  37).  In  the  “response  and  completion  of  to  service  calls”  metric,  the  metric  is 
understandable  and  quantifiable ;  it  is  based  upon  the  length  of  time  to  respond  and 
complete.  The  metric  is  also  cost  effective  because  it  is  not  labor  intensive  to  track  down 
and  is  easily  maintained  (in  a  previously  established  data  system,  WIMS).  The  service 
call  metric  is  proven  and  is  high  impact  because  it  shows  if  the  civil  engineering  is 
improving  in  responding  to  and  completing  the  number  of  service  calls,  which  also 
improves  quality  of  life  and  sustains  mission  capability.  The  response  time  and 
completion  time  metric  cannot  be  used  to  evaluate  quality  of  job  completed,  customer 
satisfaction,  or  budgetary  performance.  The  issuer  of  the  solicitation  must  decide  what 
aspects  of  the  function  are  important  and  if  decide  if  each  of  the  areas  must  have  their 
own  metrics. 

To  summarize  a  well-defined  metric,  one  must  have  a  defined  measure  of  unit.  A 
defined  measure  of  unit  that  is  quantifiable,  easily  obtained  and  understandable  to  all  who 
may  collect  the  information.  The  metric  must  have  a  sensor;  someone  must  be  identified 
and  have  the  ability  to  collect  and  store  the  data.  The  metric  must  also  have  a  frequency 
in  which  the  data  will  be  collected  and  analyzed,  based  upon  the  importance  and  demand 
of  the  information  being  collected.  Finally,  the  metric  should  be  cost-effective,  proven, 
and  high-  impact  to  properly  assess  the  function. 

Identifying  the  metrics  to  be  used  in  evaluating  any  process  is  not  an  easy  task. 
First,  it  is  important  to  recognize  critical  areas  that  need  to  be  evaluated.  The  critical 
areas  are  areas  deemed  important  to  the  success  of  the  organization  and  should  be 


31 


focused  around  the  mission  of  the  organization.  For  example,  the  critical  areas  of  the 
Civil  Engineering  Operations  Flight  are  identified  in  Air  Force  Instruction  32-1001  and 
will  be  outlined  in  Chapter  3,  Methodology. 

The  most  useful  metrics  to  any  organization  should  focus  on  the  critical  areas  of 
that  organization.  Therefore,  there  needs  to  be  some  identification  of  critical  areas. 
Recent  outsourcing  studies  in  the  private  sector  have  metrics  categorized  within  nine 
gauge  clusters:  finance/budget,  customer  satisfaction,  work  product  delivered,  quality, 
time/schedule,  business  value,  operational  service  levels,  human  resources,  and 
productivity  (Rubin,  1997,  8-9).  Table  6,  Oversight  Framework  for  Performance 
Metrics,  lists  and  defines  the  nine  clusters.  The  gauge  clusters  can  help  managers 
classify  mission  requirements  into  categories  and  then  design  and  use  metrics 
appropriately  in  the  oversight  of  the  requirements. 


Table  6.  Oversight  Framework  for  Performance  Metrics 


Gauge  Cluster 

Objective 

Finance/Budget 

Cost  management  and  on-cost  delivery  of 
services 

Customer  Satisfaction 

Critical  attributes  that  generate  satisfaction  with 
services  and  work  products  among  internal 
business  customers 

Work  Product  Delivered 

Quantifying  the  amount  of  service  or  work 
provided  in  a  given  time  period 

Quality 

Objective  and  measurable  aspects  of  quality  of 
services  and  products 

Time/Schedule 

Critical  service,  product,  and  project  time  frames 
and  the  ability  to  deliver  on-time 

Business  Value 

Measures  the  outsourcing  agreement’s  outcome 
attainment  from  the  financial/shareholder  view, 
external  customer/marketplace  view, 
organizational  learning  and  improvement  view, 
and  internal  process  improvement  view 

Operational  Service  Levels 

Critical  service  tempos,  availability,  and  delivery 
of  work  products 
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Human  Resources 

Changes  to  the  skill  inventory  and  internal  job 
satisfaction 

Productivity 

Efficiency  of  the  production  and  delivery  of 
work  products 

(Table  6  adapted  from  Rubin,  1997,  8-9) 


The  gauge  clusters  used  in  private  sector  outsourcing  can  be  used  as  a  starting 
point  in  the  evaluation  process  to  ensure  the  organization  undergoing  the  competitive 
sourcing  process  is  evaluating  the  critical  areas.  The  gauge  clusters  can  be  applied  to  the 
performance  standards  and  performance  measures  currently  used  in  the  competitive 
sourcing  process  to  ensure  all  aspects  of  the  functions  are  being  assessed  (i.e.,  financial, 
customer  satisfaction,  productivity). 

By  using  the  nine  gauge  clusters,  the  accepted  process  of  creating  a  performance 
measurement  program,  and  the  14  functions  of  the  Operations  Flight,  which  will  be 
identified  in  Chapter  three,  one  can  establish  an  effective  performance  measurement 
evaluation  system  to  determine  if  the  standards  and  metrics  found  within  the  competitive 
sourcing  process  are  properly  evaluating  critical  areas.  The  information  obtained  may 
provide  recommendations  to  improve  the  performance  standards  and  metrics  used  in  the 
competitive  sourcing  process  and  to  identify  areas  insufficiently  evaluated. 

Government  Performance  Measurement  Systems 

The  Government  has  previously  used  performance  measurement  programs.  Two 
of  the  programs  that  will  be  discussed  are  the  Total  Quality  Management  (TQM)  and  the 
Government  Performance  and  Results  Act  (GPRA).  Each  program,  TQM  and  GPRA, 
will  be  briefly  outlined  in  the  following  sections.  The  outlines  will  be  followed  by  a 
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summary  of  how  the  features  of  these  two  programs,  along  with  the  information  from 
performance  metric  design,  will  assist  in  this  research  effort. 

Total  Quality  Management 


The  Air  Force  endorsed  performance  measurement  with  Total  Quality 
Management  (TQM).  The  push  for  TQM  in  the  private  sector  began  in  the  1980’s  and 
the  government  followed  shortly  after  and  established  TQM  metrics  through  the  early 
1990’s.  The  features  of  TQM  that  are  focused  on  are:  customer  satisfaction,  employee 
involvement  and  continuous  improvement  (Cohen,  1993,  xii-xv).  TQM  was  an  initiative 
focused  on  improving  the  effectiveness  and  performance  of  the  federal  government 
(USGAO-4,  1999,  3). 

Establishing  performance  measures  to  evaluate  the  progress  made  by  TQM 
initiatives  is  a  vital  process  within  TQM  itself  The  eights  steps  of  TQM  are  identified  in 
Table  7,  Total  Quality  Management  Steps.  As  can  be  seen  in  step  seven,  performance 
measures  must  be  established  to  determine  improved  performance.  Generally,  the  steps 
involve  identifying  key  outputs  and  customers,  identifying  process  areas  creating  delays 
or  defects,  instituting  changes,  and  finally  measuring  the  results  and  comparing  them  to 
what  the  customers  really  desired  from  the  process. 


Table  7.  Total  Quality  Management  Steps 


Step 

Objective 

1 

Identify  what  processes  need  improving;  begin  with  identifying  customers  and  their  needs 

2 

Describe  the  steps  taken  in  performance  of  the  work 
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3 

Identify  the  parts  of  the  process  where  defects,  delays,  or  rework  occur  frequently 

4 

Identify  the  causes  for  the  defects,  delays,  or  rework 

5 

Improve  process  by  experimenting  with  small-scale  pilot  projects 

6 

Based  on  positive  pilot  project  results,  institute  new  procedure 

7 

Continually  monitor  the  new  process  to  ensure  it  improves  performance  over  time 

8 

Repeat  steps  1  through  7  to  continually  improve  processes 

(Cohen,  1993,  6) 


Government  Performance  and  Results  Act 


As  a  result  of  the  pursuit  for  improved  efficiency  and  effectiveness,  and  to 
continue  the  concepts  of  TQM,  the  GPRA  was  passed  in  1993.  The  GPRA  required 
Government  agencies  to  set  goals,  measure  the  performance  while  seeking  goal,  and 
report  their  results  (USGAO-3,  1996,  1).  Much  of  the  goal  setting  and  performance 
measurement  is  similar  to  that  found  in  TQM.  The  shift  resulting  from  the  initiation  of 
the  GPRA  was  from  staffing  and  activity  levels  to  results  (USGAO-3,  1996,  1).  Figure 
3,  Government  Performance  and  Results  Act,  displays  the  concept  and  steps  taken  within 
the  GPRA  to  focus  on  results.  The  three  main  steps  involve  defining  the  mission  and 
outcomes  desired,  measurement  of  the  performance,  and  utilization  of  the  performance 
information  to  improve  the  process.  Practices  9-12,  reinforcement  of  GPRA  principles, 
apply  throughout  the  use  of  the  GPRA. 
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Reinforce  GPRA  Implementation 
Practices: 

9.  Devolve  decision  making  with 
accountability 

10.  Create  Incentives 

11.  Build  expertise 

12.  Integrate  management  reforms 


Step  2 

Measure  Performance 
Practices: 

4.  Produce  measures  at  each 
organizational  level  that 

-  demonstrate  results, 

-  are  limited  to  the  vital  few, 

-  respond  to  multiple  priorities,  and 

-  link  to  responsible  programs 

5.  Collect  data 


Within  GPRA,  four  characteristics  of  successful  performance  measures  were 
identified.  The  four  characteristics  are:  demonstrate  results,  limited  to  vital  few 
(measuring  a  few  critical  areas),  respond  to  multiple  priorities,  and  link  to  responsible 
programs.  The  characteristics  are  listed  in  Table  8,  Characteristics  of  Successful 
Performance  Measures,  with  their  respective  objectives. 
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Table  8.  Characteristics  of  Successful  Performance  Measures 


Characteristic 

Objective 

Demonstrate  Results 

“Performance  measures  should  tell  each  level  how 
well  it  is  achieving  its  goals”  (USGAO-3,  1996, 

24). 

Limited  to  vital  few 

“The  number  of  measures  for  each  goal  at  a  given 
level  should  be  limited  to  the  vital  few.  Those 
vital  few  should  cover  the  key  performance 
dimensions  that  will  enable  an  organization  to 
assess  accomplishments,  make  decisions,  realign 
processes,  and  assign  accountability”  (USGAO-3, 
1996,  25). 

Respond  to  multiple  priorities 

Performance  measurements  must  take  into  account 
all  competing  interests:  quality,  cost,  customer 
satisfaction,  stakeholder  concerns,  and  other 
factors  (USGAa3,  1996,  25). 

Link  to  responsible  programs 

“Performance  measures  should  be  linked  directly 
to  the  offices  that  have  responsibility  for  making 
programs  work”;  helps  to  reinforce  accountability 
and  helps  managers  to  strive  for  goals  (USGAO-3, 
1996,  25). 

The  features  of  TQM  and  GPRA  that  will  be  used  in  this  research  are:  identify 
critical  areas,  list  the  groups’  objectives,  key  outputs,  key  customers,  produce 
measurements,  report  information  and  use  information.  When  applying  TQM  and 
GPRA,  objectives  and  goals  of  the  squadrons  can  be  identified  from  appropriate  Air 
Force  Instructions.  Next,  the  relevant  metrics  and  standards  that  coincide  with  the 
objective  and  goals  will  be  extracted  from  the  documents  and  used  in  the  evaluation. 
Then,  the  gauge  clusters  can  be  applied  to  the  functions  to  determine  what  the  major 
categories  or  clusters  the  function  is  supporting. 

The  features  of  both  TQM  and  GPRA  provide  an  initial  insight  into  identifying 
critical  mission  requirements,  which  leads  to  identifying  key  customers  and  outputs,  and 
finally,  building  metrics  to  evaluate  performance.  The  GPRA  can  be  used  further  to 
assess  the  metrics  and  standards;  the  four  characteristics  of  the  performance  measures 
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found  in  Table  8,  Characteristics  of  Successful  Performance  Measures,  along  with  the 
perfonnance  measurement  data  in  the  Creating  Performance  Metrics  section  can  be  used 
to  evaluate  the  extracted  metrics  and  standards.  Improperly  designed  standards  and 
metrics  will  be  identified  along  with  critical  areas  not  sufficiently  evaluated. 

Summary 

This  chapter  has  discussed  the  following  topics:  outsourcing  and  the  A-76 
process,  cost  comparison  process,  Performance  Work  Statements  (PWS),  Quality 
Assurance  Surveillance  Plan  (QASP),  perfonnance  metric  design.  Total  Quality 
Management  (TQM),  and  the  Government  Performance  and  Results  Act  (GPRA). 
Chapter  3,  Methodology,  will  provide  the  steps  taken  in  identifying  the  metrics  and 
standards  to  be  used  in  the  evaluation  and  will  also  identify  the  steps  taken  to  create  the 
evaluation  system  used  to  evaluate  the  metrics  and  standards. 
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3.  METHODOLOGY 


This  chapter  introduces  the  competitive  sourcing  efforts  within  the  Air  Force  and 
then  focuses  on  the  competitive  sourcing  efforts  specific  to  civil  engineering.  The 
Objectives  of  the  Civil  Engineering  Operations  Flight  are  introduced  along  with  metrics 
currently  used  to  evaluate  the  performance  of  the  Operations  Flight.  An  introduction  is 
then  given  to  the  methodology  used  to  collect  and  evaluate  the  performance  standards 
and  metrics  used  by  the  Air  Force  in  competitive  sourcing  efforts.  The  chapter  begins  by 
discussing  the  competitive  sourcing  efforts  specific  to  the  Air  Force  and  then  focuses 
specifically  on  Air  Force  Civil  Engineering,  followed  by  an  outline  of  the  criteria  used  in 
selecting  the  Air  Force  bases  for  this  research  effort.  The  PWS  and  QASP  templates  are 
introdueed,  as  is  a  general  outline  of  the  information  collected  in  the  evaluation  of 
performance  standards  and  metrics.  Then  the  steps  to  create  the  performance  standard 
and  metrie  evaluation  system  are  outlined,  leading  to  an  established  evaluation  system 
used  in  this  research  effort. 

Air  Force  Competitive  Sourcing  Efforts 

The  Air  Force  has  conducted  1,433  competitive  sourcing  competitions  since  1979, 
resulting  in  a  reported  savings  of  over  $10  Billion  and  a  maipower  reduction  of  over 
38,661  Full  Time  Equivalents  (FTE),  which  is  normally  comparable  to  one  employee 
(AFMIA-2,  2001,  n.  pag.).  Because  of  the  large  volume  of  competitive  sourcing 
activities  within  the  Air  Force,  it  was  decided  that  the  Air  Force  competitive 
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sourcing  efforts  would  provide  a  sufficient  number  of  documents  to  evaluate  and  would 
also  provide  enough  diversity  within  the  standards  and  metrics.  Therefore,  not  all 
Department  of  Defense  agencies  were  used  for  this  research  effort.  Also,  only  Air  Force 
documents  were  used  because  they  are  similar  to  one  another;  reducing  the  confusion 
from  documents  obtained  from  other  Government  agencies.  The  language  used  in  the 
Army,  Navy,  and  Marine  Corps  documents  varies  slightly  (for  instance.  Direct  Scheduled 
Work  and  the  work  order  process  are  specific  to  the  Air  Force)  and  each  service  has 
separate  data  collection  and  storage  systems,  therefore,  for  simplicity,  only  Air  Force 
documents  were  used.  Some  of  the  Air  Force  commercial  activities  initiatives  include: 
civil  engineering,  range  operations,  communication  functions,  personnel  services, 
software  programming,  supply  and  transportation,  transient  aircraft  maintenance,  airfield 
maintenance,  and  food  services. 

Air  Force  Civil  Engineering 

As  of  March  2001,  of  the  1,433  Air  Force  competitions,  the  civil  engineering  field 
had  35  commercial  activity  initiatives  in-progress  (a  service  provider  has  not  yet  been 
selected)  and  461  commercial  activity  initiatives  completed  (the  selected  service  provider 
is  either  MEO  or  private  sector  contractor).  The  civil  engineering  commercial  activity 
initiatives  include:  family- housing  maintenance,  refuse  collection,  paint  shop,  facility 
maintenance,  utility  maintenance  and  operation,  site  maintenance,  to  list  a  few. 

The  Air  Force  Civil  Engineering  field  was  the  focus  in  this  study  because  many  of 
the  civil  engineering  (CE)  activities  have  been  classified  as  a  commercial  activity  and 
have  been  identified  as  competitive  sourcing  candidates.  The  CE  field  was  the  only  field 
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evaluated  in  this  effort  to  reduce  complications  of  evaluating  across  numerous  fields 
(e.g.,  communications  and  maintenance).  From  the  list  of  all  Air  Force  CE  competitive 
sourcing  efforts,  the  larger  efforts  (defined  in  the  Scope  of  Contract  sectbn)  were 
selected  because  of  the  large  effort  involved  in  establishing  the  PWS  and  QASP.  The 
larger  competitive  sourcing  efforts  included  CE  as  a  whole  unit  and  not  just  specific  areas 
from  within  CE  (e.g.,  paint  shop  or  material  control).  An  assumption  used  in  selecting 
bases  is  that  the  larger  efforts  at  some  of  the  bases  may  have  involved  more  ingenuity  and 
creativity  when  developing  the  PWS  or  QASP.  This  ingenuity  and  creativity  may  lead  to 
significant  differences  between  the  bases  selected.  CE  is  also  key  for  the  quality  of  life 
issues  around  an  Air  Force  base  (e.g.,  facility  maintenance).  To  ensure  quality  of  life 
improves  and  mission  is  sustained  within  the  Air  Force,  CE  was  chosen  by  this  effort  as  a 
representative  for  all  Air  Force  competitive  sourcing  efforts. 

Civil  Engineer  Operations  Flight 

Many  of  the  civil  engineering  competitive  sourcing  efforts  by  the  Air  Force 
include  the  Operations  Flight  of  the  Civil  Engineer  Squadron.  The  Operations  Flight  is 
one  of  the  eight  flights  found  within  the  Civil  Engineer  Objective  Squadron  as 
established  by  AFI  38-101,  Force  Organization.  The  primary  responsibilities  of  the 
Operations  Flight  are  to:  “ensure  Air  Force  installations  can  support  the  mission, 
maintain  real  property  facilities,  and  develop  and  implement  programs  to  improve  the 
livability  of  our  base  community”  (DAF-2,  1999,  1).  The  Operations  Flight  is  tasked 
with  fourteen  functions,  identified  from  AFI  32-1001  and  listed  in  Table  9.  Table  9  lists 
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the  function  identification  as  it  appears  in  API  32-1001,  the  function  itself,  and  then  a 
brief  reference  name  to  be  used  for  the  remainder  of  the  research. 


Table  9.  Civil  Engineer  Operations  Flight  Functions 


Function  ID 

Function  Objective 

Reference  Name 

1.1 

Operate,  maintain,  repair,  construct,  and  demolish  Air  Force 
real  property  and  real  property  installed  equipment  (RPIE)  to 
accomplish  the  mission  in  the  most  timely  and  economical 
manner 

Operate,  maintain,  and 
repair 

1.2 

Provide  trained  personnel  and  technical  expertise  to  support 

Air  Force  operations  worldwide 

Trained  personnel 

1.3 

Maintain  capability  to  respond  to  and  eliminate  any 
emergency  condition  24  hours  a  day 

Emergency  response 

1.4 

Conduct  all  activities  in  compliance  with  applicable 
environmental,  fire  and  safetv  laws,  codes,  and  directives 

Compliance 

1.5 

Provide  reliable,  cost-effective  utilities  to  meet  readiness 
requirements,  satisfy  installation  needs,  and  maintain  quality 
of  life 

Reliable  utilities 

1.6 

Provide  base  support  services  (i.e.,  pest  control,  grounds 
maintenance,  snow  removal) 

Base  support 

1.7 

Establish  quality  standards  and  feedback  mechanisms  to 
assess  performance  in  meeting  mission  requirements  and 
customers’  needs 

Quality  standards 

1.8 

Establish  a  system  to  provide  customers  the  capability  to 
accomplish  work  requirements  using  their  own  resources 

Self  help 

1.9 

Develop  and  annually  update  future  plans  for  major  work 
requirements  (roofing,  pavements,  protective  coating) 

Future  plans 

1.10 

Effectively  allocate  in-service  resources,  including  people, 
facilities,  equipment,  and  vehicles  to  meet  mission  and 
customers’  needs 

Allocate  Resources 

1.11 

Provide  customers  with  the  costs  of  work  or  services 
performed  on  their  facilities 

Provide  costs  to 
customers 

1.12 

Maintain  a  time  and  material  accounting  system  to  collect  and 
report  the  cost  of  doing  business 

Time  and  material 
accounting 

1.13 

Provide  effective  logistics  support 

Logistics  support 

1.14 

Provide  an  effective  facility  manager  program 

pAF-2,  1999,  2) 


The  fourteen  functions  listed  in  Table  9  are  critical  to  the  conditions  of  the 
facilities,  quality  of  life  initiatives,  and  continuation  of  the  Air  Force  mission  and  will  be 
used  in  this  research  effort  to  identify  critical  areas  not  evaluated  by  current  PWS 
standards  and  QASP  metrics. 
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Three  major  programs  within  civil  engineering  exist  to  support  the  fourteen 
Operations  Flight  functions:  the  Planned  Work,  the  Direct  Scheduled  Work  (DSW),  and 
Recurring  Work  Program  (RWP).  The  planned  work  program  requires  detailed  planning 
and  is  broken  into  4  priorities:  priority  1  is  mission  (would  reduce  mission  capability  if 
not  accomplished),  priority  2  is  safeguard  life  and  property  (work  needed  to  provide 
secure  work  areas  or  reduce  safety  hazards),  priority  3  is  support  (support  of  mission  of 
prevention  of  a  breakdown),  and  priority  4  is  necessary  (does  not  qualify  for  a  higher 
priority)  (DAG-2,  1999,  5). 

Unlike  the  planned  work  program,  the  DSW  program  generally  does  not  require 
much  planning  and  consists  of  3  work  classifications:  emergency  (eliminate  emergency 
condition  within  24  hours),  urgent  (work  completed  within  7  calendar  days  after  receipt 
of  materials),  and  routine  (work  completed  within  30  days  of  identifying  the  requirement 
or  receipt  of  the  material)  (DAF-2,  1999,  5).  And  finally,  the  RWP  “applies  to  real 
property,  RPIE,  or  systems  and  equipments  maintained  by  the  Base  Civil  Engineer. 
Recurring  work  consists  of  operations,  recurring  maintenance,  service  work,  and  other 
recurring  work  for  which  the  scope  and  level  of  effort  are  known  without  an  earlier  visit 
to  the  job  site  each  time  the  work  is  scheduled”  (DAF-2,  1999,  5).  Each  of  these 
programs  (RWP,  planned  work,  and  DSW)  is  significant  to  quality  of  life  of  personnel 
and  to  the  success  of  the  Air  Force  mission. 

Civil  Engineering  Metrics 

Within  the  civil  engineering  career  field,  metrics  were  established  as  early  as 
1980  to  track  performance  of  civil  engineering  squadrons.  AFCESA,  the  civil 
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engineering  support  agency,  has  a  listing  of  civil  engineering  melrics  categorized  by  their 
respective  flights.  Table  10,  Subset  of  Metrics  for  Civil  Engineering  Operations  Flight, 
contains  some  of  the  metrics  for  the  Operations  Flight  that  have  been  used  over  the  years 
to  evaluate  the  Operations  Flight.  The  complete  listing  of  Operations  Flight  metrics  can 
be  found  in  Appendix  B.  Table  10,  Subset  of  Metrics  for  Civil  Engineering  Operations 
Flight,  identifies  the  functions  that  are  being  measured  and  explains  what  data  are  used  in 
the  measurement  of  the  function.  The  range  of  values  the  flight  must  attain  to  meet  the 
requirement  is  also  identified.  The  range  establishes  an  upper  limit,  a  baseline,  and  a 
lower  limit  values.  Values  obtained  from  the  metrics  should  fall  within  this  range  bound 
by  the  upper  limit  and  lower  limit  if  the  function  is  operating  sufficiently. 

The  metrics  provided  by  AFCESA  provide  a  foundation  for  the  evaluation 
process.  The  AFCESA  metrics  may  provide  insight  into  critical  areas  that  are  currently 
not  being  sufficiently  evaluated  by  current  competitive  sourcing  standards  and  metrics. 
Differences  and  similarities  between  the  AFCESA  metrics  and  the  competitive  sourcing 
metrics  will  be  highlighted  and  recommendations  will  be  made  using  the  AFCESA 
metrics  as  a  foundation. 


Table  10.  Subset  of  Metrics  for  Civil  Engineering  Operations  Flight 


METRIC 

How  Measured? 

Lower  Limit  (LL), 
Baseline  (BL),  & 
Upper  Limit  (UL) 

Work  Complete 
Emergency  DSW 
Urgent  DSW 

Routine  DSW 

Measured  WO 

number  of  commitments,  number  completed,  number 
of  DSW  backlogged 

Look  for  trends 

Open  by  category 
Emergency 

Urgent 

Routine 

Measured  WO 

number  opened  by  category 

Seek  explanation  of 
increasing  number 
of  open  WO. 

Completed  by  category 

number  completed  by  category  by  month 

Seek  explanation  for 

Hpcrpacin a  nnmhpr 
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Emergency 

Urgent 

Routine 

Measured  WO 

decreasing  number 
of  completed  WOs. 

DSW  Responsiveness 
Emergency 

Urgent 

Routine 

total  time  to  accomnlish  work  x  100% 
total  allowed  by  category 

LL:  90% 

BE:  100% 

UL:  110% 

E;  24  hrs 

U :  5  days 

R:  30  days 

Work  Satisfaction 
Emergency  DSW 
Urgent  DSW 

Routine  DSW 

number  of  commitments  comnleted  on  time  x  100% 
number  of  total  commitments 

LL:  60% 

BL:  80% 

UL;  100% 

Scheduled  Measured 
WOs 

total  number  of  davs  to  comnlete  WOs  x  100% 
total  number  of  days  scheduled  to  complete  WOs 
(estimated  completion  date  -  start  date) 

LL:  90% 

BL:  100% 

UL;  110% 

WO  life  cycle  (receipt 
to  completion  by 
category) 

Emergency  DSW 
Urgent  DSW 

Routine  DSW 

number  of  WOs  in  different  time  frequencies  (0-30 
days,  30-60  days,  etc.)  for  routine  and  in-service  DSW 
&  WO 

Look  at  increasing 
frequency  of  WOs 
within  the  different 
time  periods. 

E:  24  hrs 

U :  5  days 

R:  30  days 

Backlog  Total  and  by 
Zone  by  category 

number  of  WOs  received  vs. 
number  of  WOs  completed 

Look  for 
explanation  of 
increasing  backlog. 

Satisfaction 

Questionnaire  asking  customers  about  service.  Use  7- 
point  Likert  scales  where  1  is  bad  service  and  7  is 
good  service.  Use  approximately  20-25  questions  and 
have  at  least  1/3  of  customers  complete  questionnaire. 

LL:  average  of 
questions  =  1 

BL:  average  of 
questions  =  4 

UL:  average  of 
questions  =  7 

(AFCESA,  2001,  n.  pag.) 


Civil  Engineering  Operations  Flight  Templates 


The  PWS  templates  will  also  be  evaluated  in  this  research.  The  PWS  templates 
are  also  available  on  the  AFCESA  web  site.  The  templates  are  available  to  guide  bases  in 
writing  a  PWS.  The  templates  contain  some  standards  and  metrics,  but  each  base  must 
adapt  the  standards  and  metrics  to  meet  its  own  needs.  The  templates  will  be  used  in  the 
evaluation  in  the  same  manner  as  the  AFCESA  metrics.  The  templates  also  provide  a 
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foundation  for  recommendations  for  improving  the  competitive  sourcing  metrics.  The 
templates  are  a  key  part  of  this  research  to  determine  where  bases  have  improved  or 
declined  in  the  quality  of  performance  standards  and  metrics.  The  evaluation  of  the 
templates  will  provide  information  if  the  documents  available  to  bases  undergoing  the 
competitive  sourcing  process  are  adequate  in  providing  metric  samples  properly 
designed,  based  upon  the  criteria  of  this  research. 

Steps  of  the  Research 

The  steps  of  this  research  are  outlined  in  Table  11,  Steps  for  Evaluating 
Competitive  Sourcing  Standards  and  Metrics.  Table  1 1  consists  of  an  objective  and 
purpose  for  each  step  and  also  a  column  that  identifies  the  research  information  (typically 
the  literature  review).  The  first  step  of  this  research  effort  is  to  create  selection  criteria  to 
identify  bases  to  collect  the  PWS  and  QASP  documents.  Step  two  is  the  data  collection 
from  the  Civil  Engineering  Operations  Flight  and  the  creation  of  a  table  with  the 
collected  information  sorted  and  classified  according  to  Operations  Flight  Function  and 
gauge  cluster.  Step  three  is  the  initial  evaluation  of  the  lines  created.  Those  standards 
and  metrics  passing  the  initial  evaluation  proceed  onto  step  four,  the  secondary 
evaluation.  Once  the  evaluations  are  complete,  step  five  evaluates  the  AFCESA 
templates  and  metrics  to  use  as  a  foundation  for  recommendations  to  improve  the 
standards  and  metrics.  More  in  depth  information  on  each  of  the  steps  in  Table  1 1  will  be 
provided  in  the  following  sections. 
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Table  11.  Steps  for  Evaluating  Competitive  Sourcing  Standards  and  Metrics 


Steps  Objective  Purpose  Location  within  this 

research 


1  Create  base  selection  criteria  Identify  bases  to  use  in  Research  effort 

research 


Data  Collection: 

a)  Identify  Civil  Engineering 
Operations  Flight  critical  areas 
based  upon  functions  found  in 
AFI  32-1001 

b)  Choose  PWS  standards  and 
QASP  metrics  that  evaluate  the 
critical  areas  and  create  lines  for 
evaluation.  Classify  lines  with 
the  specific  Operations  Flight 
Function 

c)  Identify  gauge  clusters 

d)  Create  table  of  lines  with 
objective,  threshold,  surveillance 
method.  Operations  Flight 
Function,  and  Gauge  Cluster 
identified 


a)  Identify  critical  areas, 
define  mission 

b)  Combine  standards  and 
metrics  into  one  entity  to 
evaluate  and  link  them  to 
an  Operations  Flight 
Function 

c)  The  created  lines  are 
classified  into  clusters 

d)  Identify  varying  standards 
and  metrics 


a)  Ties  to  Table  5,  Steps 
to  Creating  a 
Performance 
Measurement 
Program, 

Fig  3,  Government 
Performance  and 
Results  Act,  and 
Table  7, 

Total  Quality 
Management 

b)  Ties  to  Data 
Collection  section 

c)  Ties  to  Table  6, 
Oversight  Framework 

for  Performance 
Metrics 

d)  Research  effort 


Initial  Evaluation  of  lines 
(This  step  has  a  separate  table  for 
the  evaluation,  see  Primary  Line 
Evaluation  section) 


Secondary  Evaluation  of  lines 
(This  step  has  a  separate  table  for 
the  evaluation,  see  Secondary  Line 
Evaluation  section) 


-Identify  if  measure  of  unit  is 
defined 

-  Identify  sensor 

-  Identify  frequency 

-  Identify  if  measure  is 
quantifiable 

-  Determine  if  measures  are 
understandable 

-  Identify  if  measures  are 
high  impact 


-  Identify  if  objective  is 
measured 

-  Verify  unit  of  measure, 
sensor,  and  frequency  as  in 
step  6 

-  Verify  data  is  collected  and 
where  it  is  stored 

-  If  data  is  stored,  determine 
for  how  long 

-  Determine  if  measure  is 
cost  effective 

-  Determine  if  results  are 


Ties  to  Creating 
Performance  Metrics 
section  and  Fig  3, 
Government 

Performance  and  Results 
Act 


Ties  to  Fig  3, 
Government 
Performance  and 
Results,  Table  8, 
Characteristics  of 
Successful  Performance 
Measures,  and  Table  7, 
Total  Quality 
Management  Steps 


a)  Evaluate  AFCESA  Operations 
Flight  metrics 

b)  Evaluate  AFESA  Civil 
Engineering  Operations  Flight 
Templates 


a)  Provide  foundation  to 
implement  changes  in 
metrics 

b)  Provide  foundation  to 
implement  changes  in 
metrics 


a)  Research  effort 

b)  Research  effort 
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Step  1.  Base  Selection 


Two  sources  were  used  to  gather  the  information  on  bases  undergoing 
competitive  sourcing:  the  AFCESA  web  page  and  the  A-76  Help  Desk.  The  AFCESA 
web  page  lists  all  Air  Force  commercial  activities  (CA)  completed  or  in- progress,  totaling 
1509  CA’s  as  of  May  2001.  The  A-76  Help  Desk  maintains  a  similar  listing,  strictly  for 
civil  engineering  competitive  sourcing  efforts,  which  totals  54  as  of  May  2001. 

Together,  the  two  sources  provide  a  list  of  many  bases  having  completed  or  are  currently 
undergoing  some  portion  of  the  competitive  sourcing  process.  The  listings  provided 
names  of  bases,  locations,  MAJCOMS,  contract  size  (or  scope),  and  outcome  of  the 
completed  efforts.  1 10  of  the  1509  CA’s  were  classified  as  Base  Operating  Support 
(BOS),  Civil  Engineering  (CE),  or  Multi- Support  Function  (MSF)  initiatives,  either 
completed  or  in-progress.  The  other  CA’s  were  aircraft  maintenance,  grounds 
maintenance,  and  communications,  to  name  a  few.  Ten  Air  Force  bases  were  selected  for 
this  research  effort  from  the  110  CA’s  that  include  the  Operations  Flight.  It  was  felt  that 
ten  bases  would  supply  enough  variation  within  standards  and  metrics  and  would 
adequately  reflect  differences  found  within  the  Air  Force.  The  ten  bases  were  selected 
using  4  selection  criteria.  The  4  criteria  are:  1)  scope  of  contract,  2)  major  command  the 
base  supported,  3)  location  of  base,  and  4)  outcome  (MEO  vs.  contract  or  direct 
conversion)  and  are  discussed  below. 
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Scope  of  Contract 


Many  of  the  commercial  activities  are  solicited  together  for  a  contract  with  other 
functions,  to  include  the  civil  engineering  function.  Large  contracts  result  from 
packaging  of  functions  together.  The  packages  may  or  may  not  include  the  following 
functions:  civil  engineering,  range  operations,  communication  functbns,  personnel 
services,  software  programming,  supply  and  transportation,  transient  aircraft 
maintenance,  airfield  maintenance,  and  food  services.  The  larger  of  the  contract  scopes 
are  identified  as  BOS,  MSF,  or  CE.  The  larger  scope  contracts  were  sought  after  in  this 
research  because  the  larger  contracts  included  the  entire  civil  engineering  squadron, 
which  ensures  the  entire  Operations  Flight  is  included.  Some  smaller  contracts  only  have 
parts  of  the  Operations  Flight,  for  example,  the  paint  shop  or  water  utilities  (water 
treatment),which  are  not  desirable  in  this  research.  A  much  larger  variety  of  metrics  can 
be  found  by  evaluating  the  entire  Operations  Flight.  The  larger  scope  contracts  typically 
include  civil  engineering,  along  with  some  of  the  other  functions,  within  the  solicitation 
package.  Multi- support  function  contracts  may  or  may  not  include  civil  engineering  in 
the  solicitation  package,  but  contain  some  of  the  other  functions.  A  civil  engineering 
contract  contains  only  civil  engineering  in  the  solicitation  package  and  is  identified  as 
large  because  of  the  number  of  personnel  affected  by  the  competitive  sourcing  outcome, 
typically  more  than  200  personnel.  The  three  contracts  (BOS,  multi-support  function, 
and  civil  engineering)  are  assumed  to  be  the  larger  competitive  sourcing  efforts  for  this 
research  effort. 
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Major  Command 


Documents  from  bases  in  each  of  the  major  commands  were  solicited 
(MAJCOMS)  (to  include  a  combined  command  defined  later  in  the  paragraph)  to  obtain  a 
variety  in  standards  and  metrics.  Each  MAJCOM  has  different  missions,  requirements, 
and  approaches  to  competitive  sourcing  resulting  in  the  variety  of  standards  and  metrics. 
By  obtaining  standards  and  metrics  from  various  commands,  differences  and  similarities 
could  be  highlighted.  The  PWS  and  QASP  documents  may  contain  differences  across  the 
commands  due  to  differing  leadership  styles  during  the  creation  of  the  documents. 

The  Air  Force  has  10  MAJCOMS:  Air  Combat  Command,  Air  Education  and 
Training  Command,  Air  Force  Materiel  Command,  Air  Force  Reserve  Command,  Air 
Force  Space  Command,  Air  Force  Special  Operations  Command,  Air  Mobility 
Command,  Pacific  Air  Command,  United  Air  Force  Academy,  and  United  States  Air 
Force  in  Europe.  Each  command  was  reviewed  to  determine  the  amount  of  competitive 
sourcing  attempted  within  the  command.  Air  Combat  Command  had  much  of  the 
competitive  sourcing  effort  due  to  its  mission.  Air  Combat  Command  lent  itself  as  a 
good  competitive  sourcing  candidate  because  the  support  functions  were  not  military 
essential  and  any  money  saved  could  be  transferred  to  weapons  systems.  The  Air  Force 
Space  Command,  Air  Force  Special  Operations  Command,  Air  Mobility  Command, 
Pacific  Air  Command,  United  Air  Force  Academy,  and  United  States  Air  Force  in 
Europe  were  combined  into  one  group  for  this  research  effort  (will  be  referred  to  as  the 
“combined  command”  for  the  remainder  of  this  research  effort)  because  of  fewer 
competitive  sourcing  efforts  within  these  commands. 
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Location  of  Air  Force  Bases 


The  location  of  the  bases  was  used  as  a  selection  criterion.  Competitive  sourcing 
efforts  were  selected  for  this  research  from  a  range  of  locations  across  the  United  States 
to  capture  some  variance  that  would  be  expected  due  to  geography.  The  locations  of  the 
bases  were  broken  into  six  regions  or  areas  as  used  by  the  Air  Force  Personnel  Center 
(AFPC).  Figure  4,  Location  of  Air  Force  Bases,  displays  the  location  of  all  active  duty 
Air  Force  bases  and  their  respective  areas  used  in  this  research.  Area  4  was  the  only  area 
omitted  from  this  research  simply  because  there  were  no  bases  passing  the  other  three 
eriteria  found  in  the  Area.  Bases  not  located  on  Figure  4  are  the  Reserve  bases.  Two 
reserve  bases  were  used  in  this  research  and  are  located  in  Area  2  and  Area  3.  Overseas 
bases  were  not  used  because  of  host  nation  agreements  that  may  affect  the  competitive 
sourcing  process. 

Some  variation  may  arise  because  some  bases  are  located  next  to  cities  and  some 
are  located  in  mral  areas.  The  variation  of  the  city  size  is  not  evaluated  in  the  selection 
criteria  but  may  have  an  effect  on  the  creation  of  the  contracting  documents  (e.g.,  PWS  or 
QASP).  For  instance,  bases  located  next  to  large  cities  may  have  had  access  to  many 
private  sector  businesses  that  have  unique  methods  of  performance  evaluation.  These 
unique  methods  may  or  may  not  have  been  adapted  by  the  neighboring  base  within  their 
competitive  sourcing  documents.  In  order  to  obtain  all  possible  representations  of 
standards  and  metrics,  the  geography  of  the  base  was  used  as  a  selection. 
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Figure  4.  Location  of  Air  Force  Bases 

(Adapted  from  AFPC,  2001,  n.  pag.) 


Outcome  of  Competitive  Sourcing  Efforts. 

Finally,  competitive  sourcing  documents  from  various  bases  were  also  selected 
based  upon  the  outcome  of  the  competitive  sourcing  effort.  The  possible  outcomes  for 
the  competitive  sourcing  considered  for  this  research  effort  were:  Most  Efficient 
Organization  (MEO),  contractor,  or  direct  conversion  to  contractor.  It  was  decided  to 
obtain  a  similar  number  of  MEO  awards  as  contractor  awards.  It  was  also  decided  to 
collect  at  least  one  direct  conversion  award  to  obtain  a  variety,  and  because  there  are 
fewer  direct  conversion  awards  in  the  Air  Force,  it  was  decided  that  one  sample  would  be 
sufficient.  Some  of  the  civil  engineering  units  at  selected  bases  may  still  be  in  the  cost 
comparison  process  (pre- award  status),  but  were  selected  to  identify  changes  and  possible 
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improvements  in  performance  standards  and  metrics  within  these  newer  solicitations. 

The  planned  10  bases  selected  for  this  research  provide  enough  diversity  to  adequately 
represent  Air  Force  competitive  sourcing  documents.  The  standards  and  metrics  pulled 
from  the  competitive  sourcing  documents  will  not  be  referenced  back  to  the  specific  base 
from  which  it  came,  to  maintain  the  integrity  of  the  civil  engineering  unit. 

Step  2.  Data  Collection 

Once  the  bases  were  identified  and  the  document  collection  was  completed,  a 
series  of  steps  was  used  to  collect  the  standards  and  metrics.  The  first  step  was  to  review 
the  entire  PWS  document  for  a  service  delivery  summary,  a  table  that  lists  the  standards 
of  the  service.  This  table  is  typically  located  in  section  two  of  a  PWS,  but  can  also  be 
located  in  technical  exhibits  portion  of  the  PWS,  or  in  a  performance  management  plan, 
which  is  separate  from  a  PWS  and  specific  to  the  Air  Education  and  Training  Command. 
The  performance  management  plan  lists  the  standards  and  metrics  that  the  base  would 
like  measured  during  the  contract.  All  standards  and  metrics  were  compiled  from  the 
performance  management  plan,  service  delivery  summary,  and  the  technical  exhibits.  The 
PWS  documents  ranged  in  size  from  approximately  70  pages,  to  over  500  pages.  Once 
the  standards  were  located,  the  QASP  documents  were  reviewed  to  find  the  metrics  that 
corresponded  to  the  standards.  The  performance  management  plans,  unlike  the  longer 
PWS,  contained  one  to  six  pages  listing  the  standards  and  metrics  together. 

The  second  step  of  the  data  collection  involved  three  steps  of  assembly,  1)  to 
assemble  the  PWS  standard  and  its  corresponding  metric  into  a  “line”  that  contains  the 
objective,  threshold,  and  surveillance  method,  2)  use  Table  9,  Civil  Engineer  Operations 
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Flight  Functions  to  classify  the  lines,  and  3)  use  Table  6,  Oversight  Framework  for 
Performance  Metrics,  to  further  elassify  the  lines.  The  lines  eontain  the  standard  and 
metrie  verbatim  from  the  PWS  and  QASP  documents.  Clarifieation  (e.g.,  aeronyms)  on 
the  information  found  within  the  line  can  be  found  with  the  lines  in  Appendix  C.  Table 
12,  Line  Creation,  Step  7,  is  an  example  of  a  standard  and  metrie  used  to  establish  a  line. 
The  ereation  of  the  lines  allow  for  easier  evaluation  of  the  standards  and  metrie  s. 


Table  12.  Line  Creation,  Step  1 


Objective 

Threshold 

Surveillance 

Provide  Production  Control  that  is 
Professional  and  Courteous  at  all 
times. 

0  Defects.  Lot  is  number  of  calls 
received  or  verbal  requests  taken 
monthly. 

Customer  Complaint 

Table  13,  Line  Creation,  Step  2,  demonstrates  how  the  example  metrie  in  Table 
12  is  classified  into  one  of  the  14  Civil  Engineer  Operations  Flight  Funetions.  The 
classifications  of  the  lines  into  Civil  Engineer  Operations  Flight  Funetions  indieate  what 
areas  of  the  Operations  Flight  are  evaluated  and  to  what  extent  the  areas  are  evaluated 
(some  areas  may  have  more  than  one  metric  measuring  the  area).  The  classification  of 
the  Operations  Flight  Function  is  based  upon  the  information  contained  in  the  objeetive 
of  the  line.  The  classification  of  the  Operations  Flight  Function  is  subjective  and  based 
upon  the  experience  of  the  researcher. 
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Table  13.  Line  Creation,  Step  2 


Objective 

Threshold 

Surveillance 

Operations  Flight 
Function 

Provide  Production  Control 
that  is  Professional  and 

Courteous  at  all  times. 

0  Defects.  Lot  is  number  of 
calls  received  or  verbal 
requests  taken  monthly. 

Customer  Complaint 

1.2,  Provides  trained 
personnel  and  technical 
expertise  to  support 
operations  worldwide 

Table  14,  Sample  Line,  continues  the  classification  of  the  metric  in  Table  13.  The 
standard,  metric,  and  Operations  Flight  Function  is  classified  into  one  of  the  nine  gauge 
clusters  from  Table  6,  Oversight  Framework  for  Performance  Metrics.  The 
classifications  of  the  lines  into  gauge  clusters  determine  what  aspect  of  the  Operations 
Flight  Function  is  evaluated  (e.g.,  quality,  customer  satisfaction,  or  cost).  The  gauge 
cluster  classification  was  based  upon  information  contained  in  the  threshold  and  the 
objective  of  the  line.  The  combination  of  the  threshold  and  objective  provided  the 
direction  as  to  what  aspect  was  measured  and  what  gauge  cluster  should  be  used  to 
classify  the  line  (i.e.,  customer  satisfaction,  quality,  time/schedule).  The  classification  of 
the  gauge  cluster  is  partially  subjective. 

Once  the  list  of  standards  and  metrics  were  extracted  and  classified  by  Operations 
Flight  Function  and  gauge  cluster,  each  of  the  created  lines  was  referenced  numerically 
for  bookkeeping  purposes  throughout  this  research  effort.  See  Table  14,  Sample  Line,  for 
an  example  of  the  line  with  all  categories  listed  for  this  research.  Once  the  lines  were 
created,  they  were  divided  into  groups  according  to  their  Operations  Flight  Function 
classification. 
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Table  14.  Sample  Line 


ID 

Objective 

Threshold 

Surveillance 

Operations  Flight 
Function 

Gauge 

Cluster 

1 

Provide  Production 
Control  that  is 

Professional  and 
Courteous  at  all  times. 

0  Defects.  Lot  is 
number  of  calls 
received  or  verbal 
requests  taken  monthly. 

Customer 

Complaint 

1 .2,  Provides  trained 
personnel  and 
technical  expertise  to 
support  operations 
worldwide 

Customer 

Satisfaction 

If  an  objective  and  threshold  were  found  to  contain  two  or  more  gauge  clusters, 
for  example  time/schedule  and  quality,  the  one  objective  and  threshold  were  broken  into 
two  separate  lines,  one  for  the  time/schedule  gauge  cluster  and  another  for  the  quality 
gauge  cluster.  This  was  done  to  reduce  the  confusion  of  how  to  classify  the  line.  An 
example  of  an  objective  and  threshold  that  were  split  into  two  Hnes  is  given  in  Table  15, 
Split  Objectives  and  Threshold.  The  original  objective  was,  “Conduct  an  analysis  of  all 
work  orders  having  a  10%  or  greater  difference  between  planned  and  estimated  hours. 
Provide  a  copy  of  the  analysis  to  the  QAE,  within  one  week  of  receipt  of  Work  Order 
Variance  Report.”  Both  business  value  and  time/schedule  gauge  clusters  can  be 
identified;  therefore,  two  lines  were  created  as  seen  in  Table  15. 


Table  15.  Split  Objectives  and  Threshold 


ID 

Objective 

Threshold 

Surveillance 

Operations  Flight 
Function 

Gauge 

Cluster 

23 

Conduct  an  analysis  of  all 
workorders  having  a  10% 
or  greater  difference 
between  planned  and 
estimated  hours. 

1  Defect.  Lot  is 
number  of  completed, 
planned  work  orders 
for  the  month. 

Checklist 

1.12,  Maintains  a  time  and 
material  accounting  system 
to  collect  and  report  the 
cost  of  doing  business 

Business 

Value 

24 

Provide  a  copy  of  the 
analysis  to  the  QAE, 
within  one  week  of  receipt 
of  Work  Order  Variance 
Report 

1  Defect.  Lot  is 
number  of  completed, 
planned  work  orders 
for  the  month. 

Checklist 

1.12,  Maintains  a  time  and 
material  accounting  system 
to  collect  and  report  the 
cost  of  doing  business 

Time/ 

Schedule 
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Standards  and  metrics  that  were  site  specific  were  not  analyzed  in  this  research. 
The  site- specific  standards  and  metrics  are  for  buildings  that  were  located  only  at  the 
bases  that  were  selected.  For  example,  one  base  had  a  large  hospital  and  had  standards 
and  metrics  that  applied  strictly  to  it.  Those  standards  and  metrics  were  not  included  in 
the  evaluation  because  this  research  effort  seeks  standards  and  metrics  that  are  found  Air 
Force  wide.  Other  examples  of  standards  and  metrics  not  used  in  the  evaluation  are 
regarding  a  weather  agency,  ammunition  storage  facilities,  and  a  headquarters  building. 
Because  these  facilities  are  not  typically  found  at  every  base,  they  were  omitted  from  the 
evaluation. 

To  evaluate  the  lines  for  proper  metric  design,  an  evaluation  system  was 
established.  This  system  was  created  from  the  various  metric  design  literatures,  features 
from  Total  Quality  Management  and  Government  Performance  and  Results  Act,  and 
current  civil  engineering  metrics.  The  evaluation  system  was  then  applied  to  each  of  the 
lines. 

Step  3.  Primary  Line  Evaluation 

The  Primary  Line  Evaluation  is  a  method  established  by  this  research  to  test  each 
of  the  created  lines  to  determine  if  the  standards  and  metrics  within  the  lines  are  properly 
designed.  Each  line  created  from  the  PWS  and  QASP  documents  was  evaluated  using 
Table  16,  Primary  Line  Evaluation  Table.  Specific  features  of  the  performance  metric 
design  literature,  the  Total  Quality  Management  process,  and  the  Government 
Performance  and  Results  Act  will  be  identified  to  use  in  the  primary  evaluation.  The  six 
criteria  contained  in  the  primary  evaluation  step,  shown  in  Table  16  are:  1)  defined 
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measure  of  unit,  2)  sensor,  3)  frequency,  4)  understandable,  5)  quantifiable,  and  6)  high 
impact.  Buchheim  developed  criteria  1,  2,  and  3  and  Edberg  developed  criteria  4,  5,  and 


6,  both  are  discussed  in  the  Creating  Performance  Metrics  section.  The  six  criteria  tie 
back  to  TQM  and  GPRA  because  the  criteria  indicate  that  a  measurement  system  is  in 
place  to  evaluate  the  process  (as  specified  by  TQM  and  GPRA  principles). 

The  primary  evaluation  system  inc  ludes  a  “yes/no”  block  to  identify  if  the 
researcher  felt  the  line  passed  or  failed  the  criteria  (“no”  indicates  a  fail).  Finally,  the 
evaluation  system  has  a  justification  block  to  allow  the  researcher  to  explain  the  given 
line’s  success  or  failure  for  each  criterion. 


Table  16.  Primary  Line  Evaluation  Table 


Criteria 

Yes/No 

Justification 

Defined  unit  of  measure? 

Sensor? 

Frequency? 

Understandable? 

(Not  difficult  to  understand) 

Quantifiable? 

(Reduced  personal  infiuence 
or  judgment) 

High  Impact? 

(Affect  Quality  of  Life, 

Mission,  or  Customer) 

Step  4.  Secondary  Line  Evaluation 


A  secondary  line  evaluation  was  applied  to  all  metric  and  standard  lines  that 
passed  the  primary  evaluation.  The  secondary  line  evaluation  purpose  was  to  perform  a 
quantitative  evaluation  of  the  metrics  and  standards  by  using  specific  information  from 
the  base.  Representatives  from  the  bases  were  asked  to  input  information  regarding  all 
lines  that  passed  the  primary  line  evaluation  (step  3)  because  certain  aspects  of  this 
research  were  not  easily  accomplished  by  simply  reviewing  the  lines.  If  a  line  passed  the 
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primary  line  evaluation  by  receiving  a  “yes”  on  the  six  criteria,  personnel  from  the  base 
from  which  the  line  came  from  was  asked  to  answer  the  questions  found  in  the  secondary 
line  evaluation,  shown  in  Table  17.  The  five  criteria  contained  in  the  secondary 
evaluation  are:  1)  is  objective  measured,  2)  where  is  collected  information  stored,  3)  how 
long  is  the  information  stored,  4)  cost  effective,  and  5)  proven.  Criteria  1,  2,  and  3  are 
developed  from  TQM  and  GPRA,  which  are  located  in  the  Government  Performance 
Measurement  Systems  section,  and  criteria  4  and  5  are  developed  from  Edberg  and  are 
located  in  the  Creating  Performance  Metrics  section.  Criteria  1,  2,  and  3  tie  back  to 
TQM  and  GPRA  because  the  criteria  indicate  that  a  measurement  system  is  in  place  and 
is  the  information  is  used  to  evaluate  the  process  (as  specified  by  TQM  and  GPRA 
principles). 

The  secondary  evaluation  system  includes  a  “yes/no”  block  to  identify  if  the  base 
personnel  felt  the  line  passed  or  failed  the  criteria  (“no”  indicates  a  fail).  Finally,  the 
evaluation  system  has  a  justification  block  to  allow  the  researcher  to  explain  the  given 
line’s  success  or  failure  for  each  criterion. 


Table  17.  Secondary  Line  Evaluation 


Criteria 

Yes/No 

Justification 

Is  objective  measured? 

Where  is  collected  information  stored? 
(Accessible  to  those  that  need  information) 

How  long  is  the  collected  information  stored? 

Cost  Effective? 

(Value  of  obtained  information  outweighs  cost 
of  seeking  information) 

Proven? 

(Has  shown  demonstrated  results) 

Table  17  was  only  applied  to  the  lines  that  passed  all  six  criteria  found  in  the 
primary  evaluation  (Table  16).  By  evaluating  the  lines  with  Table  16  and  Table  17,  each 
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standard  and  metric  was  evaluated  for  proper  design  and  determined  if  it  was  effective  or 
not  effective  as  found  in  the  justification  portion  of  the  evaluations,  resulting  in 
Operations  Flight  functions  being  identified  as  having  sufficient  or  insufficient 
performance  measures. 

The  result  of  the  primary  and  secondary  evaluations  will  provide  standards  and 
metrics  that  are  properly  (or  improperly)  designed.  Those  standards  and  metrics  failing 
some  portion  of  the  primary  ard  secondary  evaluations  will  be  listed  and  reviewed  to 
determine  recommendations  for  improvement. 

Step  5,  AFCESA  Operations  Flhht  Metrics  and  Template  Evaluations 

The  final  step  in  the  evaluation  process  was  to  examine  the  AFCESA  Operations 
Flight  competitive  sourcing  template  and  AFCESA  Operations  Flight  metrics  for  positive 
metrics  and  standards  that  were  not  included  in  the  PWS  and  QASP  documents  from  the 
bases.  The  templates  and  metrics  are  accessible  to  all  bases  and  were  evaluated  because 
they,  1)  provided  additional  standards  and  metrics  that  evaluated  different  critical  areas, 
2)  provided  standards  and  metrics  that  were  designed  differently  than  the  PWS  and 
QASP  standards  and  metrics,  and  3)  had  not  been  previously  evaluated  for  proper  design 
using  this  research  methodology.  The  evaluation  was  conducted  in  the  same  manner  as  it 
was  for  the  161  standard  and  metric  lines.  Each  metric  was  listed  separately  and  then 
classified  according  to  its  Operations  Flight  Function  and  gauge  cluster.  The  primary 
evaluation  was  then  applied  to  each  metric. 
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Operations  Flight  Metrics 


The  important  features  of  the  Subset  of  Metrics  for  Civil  Engineering  Operations 
Flight,  located  in  Table  10,  used  in  this  research  are  the  metric  itself  and  how  it  is 
measured;  they  provide  the  objective  and  the  surveillance  information.  Each  of  the  47 
metrics  was  evaluated  to  determine  if  current  standards  and  metrics  evaluated  a  critical 
area.  For  instance,  the  Direct  Scheduled  Work  (DSW)  responsiveness  found  in  Table  10 
evaluates  the  DSW  responsiveness,  which  has  been  identified  as  a  critical  area.  The 
metrics  were  then  classified  according  to  Operations  Flight  Functions  and  gauge  clusters. 
The  primary  line  evaluation  was  then  applied  to  each  of  the  47  metrics.  Critical  areas 
lacking  sufficient  evaluation  were  identified  by  failed  (“no”  response)  response  to  the 
questions  from  the  primary  line  evaluation.  The  Operations  flight  metrics  also  provided  a 
foundation  for  this  research  to  create  quality  metric  s  to  be  used  in  future  competitive 
sourcing  efforts. 

Templates 

Similarly  for  the  templates,  the  nine  metrics  were  classified  according  to 
Operations  Flight  Functions  and  gauge  clusters.  Each  of  the  nine  metrics  was  then 
evaluated  with  the  primary  line  evaluation.  Critical  areas  lacking  sufficient  evaluation 
were  identified  by  a  failed  response  to  a  primary  line  evaluation  question.  The  metrics 
found  within  the  templates  provided  an  avenue  to  identify  areas  not  sufficiently  evaluated 
in  competitive  sourcing  efforts.  The  template  metrics  also  provided  a  foundation  for  this 
research  to  create  quality  metrics. 
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Summary 


This  chapter  outlined  the  methodology  that  was  used  in  the  standard  and  metric 
evaluation  process.  First,  Air  Force  competitive  sourcing  efforts  were  introduced  and 
then  the  civil  engineering  competitive  sourcing  efforts  were  identified  as  the  relevant 
field  of  study.  Then,  civil  engineering  metrics  and  the  PWS  templates  were  discussed. 
The  research  steps  were  then  introduced,  as  were  the  base  selection  criteria.  Then,  the 
creation  of  the  performance  standard  and  metric  lines  were  outlined,  followed  by  the 
steps  for  creating  the  primary  and  secondary  evaluation  systems.  Finally,  the  role  of 
AFCESA  templates  and  Operations  Flight  metrics  in  the  research  was  outlined.  Chapter 
4,  Analysis,  will  detail  the  analysis  and  provide  the  results  from  the  evaluation  of  the  161 
standard  and  metric  lines,  the  AFCESA  Operations  Flight  metrics,  and  the  AFCESA 
Operations  Flight  competitive  sourcing  templates. 


62 


4.  ANALYSIS  AND  RESULTS 


Chapter  4  presents  the  development  and  analysis  of  the  lines  containing  the 
performance  standards  and  metrics  and  the  results  from  those  analyses.  First,  a 
discussion  of  how  the  information  was  collected  and  categorized  will  be  presented.  Then, 
the  primary  line  evaluation  will  be  presented  in  the  Primary  Line  Evaluation  section,  the 
secondary  line  evaluation  will  be  presented  in  Secondary  Line  Evaluation  section,  and 
then  the  evaluation  of  the  AFCESA  metrics  and  templates  will  be  provided  in  the 
Evaluation  of  Operations  Flight  PWS  Templates  and  Operations  Flight  Metrics  section. 
Finally,  the  19  proposed  Operation  Flight  metrics  will  be  presented.  In  closing,  the 
results  of  the  evaluations  will  be  discussed. 

Information  Collection 

The  best  method  to  obtain  the  names  of  bases  either  having  completed  or  initiated 
the  competitive  sourcing  process  was  from  the  “camreports”  found  on  the  AFCESA 
website,  which  lists  all  completed  competitive  sourcing  efforts  (1433)  and  the 
competitive  sourcing  efforts  which  are  underway  (76)  in  the  Air  Force.  The  information 
is  categorized  in  completed  initiatives  and  initiatives-under-progress  groupings.  The 
AFCESA  listing  also  groups  the  competitive  sourcing  efforts  alphabetically  by  name 
(name  of  effort,  family  housing,  grounds  maintenance,  supply,  and  so  on)  and  by 
MAJCOM.  Additional  information  on  the  AFCESA  listing  is  the  type  of  contract  (BOS, 
Multi-support  Function  [MSF],  or  Civil  Engineering  [CE]),  the  number  of  affected 
positions,  and  the  outcome  of  the  contract  (contractor  or  MEO). 
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During  this  research  effort,  the  AFCESA  website  was  being  upgraded,  therefore 
new  data  were  not  being  input  in  the  AFCESA  listing.  New  or  under- way  competitive 
sourcing  efforts  were  gathered  from  the  A-76  Help  Desk,  because  they  maintained  their 
separate  files  of  civil  engineering  competitive  sourcing  efforts.  The  A-76  Help  Desk  files 
contained  the  name  of  the  base,  name  of  contract,  number  of  positions  affected,  and  a 
short  description  (when  completed,  in-progress,  and  outcome).  At  the  time  of  this 
research  the  A-76  Help  Desk  files  contained:  completed  initiatives  (30),  canceled 
initiatives  (5),  in- progress  initiatives  (16),  and  on-hold  initiatives  (3).  Of  these  54 
initiatives  listed  by  the  A-76  Help  Desk,  only  the  completed  or  in-progress  initiatives 
were  considered  for  this  research. 

From  the  two  listings  (AFCESA  web  site  and  A-76  Help  Desk),  ten  bases  were 
chosen  as  a  representative  sample  of  Air  Force  competitive  sourcing  documents.  The 
identification  of  bases  included  sorting  by  the  selection  criteria:  Scope  of  Contract, 
Location  of  Base,  MAJCOM,  and  Outcome.  Table  18,  Selected  Bases,  identifies  the 
bases  along  with  the  base  selection  characteristics.  A  goal  of  this  research  was  to  obtain 
a  variety  of  bases,  and  the  ten  bases  do  provide  the  variety  based  upon  the  selection 
criteria.  The  three  scopes  (BOS,  MSF,  and  CE)  are  represented,  each  command  is 
represented  (combined  command  is  compiled  of  several  commands),  the  bases  are 
geographically  separated,  and  each  of  the  outcomes  is  also  represented. 


64 


Table  18.  Selected  Bases 


*Note:  PWS  and  QASP  documents  are  highlighted  in  the  Bibliography 


Base 

Command 

Location 

Outcome 

Cheyenne  Mountain 

CE 

Combined  Command 

Area  5 

Contract 

Goodfellow 

BOS 

Air  Education  and  Training  Command 

Area  6 

MEO 

Grissom 

BOS 

Air  Force  Reserve 

Area  3 

Contract 

Keesler 

MSF 

Air  Education  and  Training  Command 

Area  1 

Pre -award 

Kirtland 

CE 

Air  Force  Materiel  Command 

Area  5 

Direct  Conversion 

Maxwell 

MSF 

Air  Education  and  Training  Command 

Area  1 

Contract 

Offutt 

BOS 

Air  Combat  Command 

Area  3 

Pre -award 

Vandenberg 

BOS 

Combined  Command 

Area  5 

Contract/MEO 

Westover 

BOS 

Air  Force  Reserve 

Area  2 

Contract 

BOS 

Air  Force  Materiel  Command 

Area  2 

MEO 

The  lines  that  were  ereated  by  the  performance  standards  and  metrics  were  not 
referenced  back  to  the  individual  bases.  A  goal  of  this  research  was  to  obtain  examples 
of  metrics  and  standards  (good  or  bad)  and  not  to  identify  problems  with  specific  bases. 

The  information  for  the  outcome  portion  of  Table  18  is  as  depicted  in  the 
AFCESA  camreports.  After  contacting  the  bases,  it  was  discovered  there  were  some 
errors  in  the  listings.  For  example,  Vandenberg  AFB  was  listed  as  having  a  BOS 
contract,  but  they  have  four  separate  contracts  for  pieces  of  their  civil  engineering  unit. 
Two  of  the  contracts  for  Vandenberg  civil  engineering  went  to  contractors,  while  the 
other  two  went  to  the  MEO.  Due  to  the  separation  of  contracts,  Vandenberg  does  not 
have  a  BOS  contract.  Therefore,  Vandenberg  was  not  included  in  this  research  effort. 
Also,  Keesler  AFB  had  withdrawn  their  documents  from  solicitation,  and  subsequently, 
Keesler  was  also  removed  from  this  research  effort  because  the  documents  were  not 
complete  at  the  time  of  this  research.  The  remaining  eight  bases  provided  their  PWS  and 
QASP  documents  for  this  research. 

Each  of  the  eight  PWS  documents  were  reviewed  and  standards  found  within  the 
Service  Delivery  Summary  or  Technical  Exhibits  were  listed  in  a  table.  The  eight  QASP 
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documents  were  then  reviewed  to  match  the  metrics  within  the  documents  to  their 
respective  standards  to  create  the  lines  for  evaluation.  For  Maxwell  AFB,  the  standards 
and  metrics  were  found  within  their  Performance  Management  Plan.  The  Performance 
Management  Plan  is  an  effort  by  the  Air  Education  and  Training  Command  to  design  and 
enforce  performance  metrics.  The  plan  details  what  information  to  collect,  when  to 
collect,  and  how  to  interpret  the  data.  Maxwell  was  the  only  base  used  in  this  research 
that  used  a  Performance  Management  Plan. 

Created  Standard  and  Metric  Lines 

There  were  a  total  of  161  standard  and  metric  lines  extracted  from  the  8  base 
documents  that  contained  the  objective,  threshold,  and  surveillance.  The  161  lines  were 
further  classified  according  to  the  14  Operations  Flight  Functions  and  the  nine  gauge 
clusters.  Table  19,  Results  of  the  Classifications,  summarizes  the  numbers  found  in 
regards  to  the  Operations  Flight  Functions  and  the  gauge  clusters.  Table  19  lists  all  the 
Operations  Flight  Functions  by  their  reference  name  from  Table  9,  Civil  Engineering 
Operations  Flight  Functions,  and  only  lists  the  gauge  clusters  found  in  the  classifications. 
There  are  nine  gauge  clusters,  but  only  seven  were  found  during  the  classification  of  the 
161  lines. 

Table  19  summarizes  the  findings  of  the  classifications.  For  example,  within  the 
161  created  lines,  38  lines  were  identified  as  Operations  Flight  Function  1.1,  Operate, 
Maintain,  and  Repair,  signifying  an  emphasis  on  this  particular  function  by  the  bases  that 
were  evaluated.  The  Operational  Service  Level  was  the  most  frequent  gauge  cluster 
classification  of  the  161  lines.  At  the  opposite  end,  there  are  3  Operations  Flight 
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Functions  (Operations  Flight  Function  1.9,  Future  plans,  Operations  Flight  Function  1.11, 
Provide  costs  to  customers,  and  Operations  Flight  Function  1.14,  Facility  manager 
program)  with  no  metric  evaluations  identified  from  the  161  lines.  Also,  there  are  only  2 
and  3  gauge  clusters  identifying  customer  satisfaction  and  work  product  delivered, 
respectively,  for  the  161  lines  evaluated.  Areas  that  lack  sufficient  evaluation  can  be 
readily  identified  if  there  are  no  standards  and  metrics  to  evaluate  them.  Also,  the 
increased  number  of  metrics  located  within  a  certain  area  identifies  areas  where  metrics 
have  been  concentrated. 

The  161  lines  with  their  Operation  Flight  Function  and  gauge  cluster 
classifications  can  be  found  in  Appendix  C.  From  the  161  lines,  a  subset  of  each 
Operations  Flight  Function  was  created  to  simplify  the  evaluation  process.  Each  of  the 
lines  with  its  primary  line  evaluation  table  (see  Primary  Line  Evaluation  section)  can  be 
found  in  Appendices  D  through  N,  divided  into  their  Operations  Flight  Functions. 
Operations  Flight  Functions  classified  as  1.1  are  found  in  Appendix  D,  Operations  Flight 
Functions  classified  as  1.2  are  found  in  Appendix  E,  and  so  on.  There  were  no 
Operations  Flight  Functions  classified  as  1.9,  1.1 1,  or  1.14,  therefore,  there  are  no 
appendices  for  them  and  Operations  Flight  Functions  1.10,  1.12,  and  1.13  are  found  in 
Appendices  L,  M,  and  N,  respectively. 
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Operation  Flight  Functions 

1.1,  Operate,  maintain,  and  repair 

1 .2,  Trained  personnel 

1 .3,  Emergency  response 

1 .4,  Compliance 

1.5,  Reliable  utilities 

1.6,  Base  support 

1 .7,  Quality  standards 

1.8,  Self  help 

1 .9,  Future  plans 

1.10,  Allocate  Resources 

1.11,  Provide  costs  to  customers 

1.12,  Time  and  material  accounting 

1.13,  Logistics  support 

1.14,  Facility  manager  program 

Total  number  of  metrics  found 
within  gauge  cluster  classification: 
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Primary  Line  Evaluation 


Once  the  161  lines  were  divided  into  their  respective  Operations  Flight  Functions, 
an  evaluation  was  used  to  determine  if  the  lines  were  properly  designed.  Using  metric 
design,  TQM,  and  GPRA  features  identified  in  Chapter  II,  a  series  of  questions  was 
developed:  1)  is  there  a  defined  unit  of  measure?  2)  Is  there  a  sensor?  3)  Is  there  a 
frequency?  4)  Is  the  line  understandable?  5)  Is  the  line  quantifiable?  and  6)  Is  the  line 
high  impact?  The  answers  to  these  six  questions  for  each  line  is  based  upon  information 
obtained  from  the  competitive  sourcing  documents,  and  researcher  knowledge  of  the 
documents.  Each  of  the  161  lines  created  from  the  PWS  and  QASP  documents  were 
evaluated  with  the  same  six  questions;  a  question  resulting  in  a  “no”  answer  indicated  a 
fail,  and  a  “yes”  indicated  a  pass  for  each  of  the  six  questions.  For  simplicity,  the 
evaluation  questions  are  exclusive,  if  a  line  failed  question  #1  and  question  #3  during  the 
primary  evaluation;  the  line  was  classified  as  only  failing  question  #1  and  was  only 
discussed  as  failing  #1  (it  was  not  discussed  as  failing  question  #3). 

Of  the  161  lines,  7  lines  failed  because  there  was  no  defined  unit  of  measure,  for 
example  time,  cost,  or  work  order.  None  of  the  lines  failed  the  sensor  question;  each  line 
had  some  method  identified  to  collect  the  data  for  the  standard  or  metric.  3 1  lines  failed 
because  there  was  no  established  frequency  within  the  line;  there  was  a  lack  of  a  time 
period  in  which  to  collect  the  information  (weekly,  monthly,  or  yearly).  61  lines  failed 
because  they  were  not  understandable;  there  was  a  mismatch  between  the  objective, 
threshold,  or  surveillance.  1 1  lines  failed  because  they  were  not  quantifiable,  many  of  the 
lines  not  quantifiable  were  subjective  and  could  not  have  specific  or  rigid  measurement. 
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Finally,  18  lines  failed  because  they  were  not  high  impact;  the  lines  were  evaluated  from 
the  base  personnel  as  the  customer,  not  the  MAJCOM  or  Civil  Engineer  unit. 

Identification  of  the  customer  is  critical  in  determining  what  is  important  to  the 
Civil  Engineer  unit  for  performance  measurement  purpose  and,  for  example,  what  is 
important  to  the  MAJCOM  may  not  be  important  to  the  customer.  Examples  for  each  of 
the  six  primary  evaluation  questions  will  be  presented  later  in  the  chapter.  The  list  of 
lines  passing  the  primary  evaluation  can  be  found  in  Appendix  O  along  with  their 
respective  secondary  line  evaluation.  For  the  secondary  evaluation,  all  lines  were  listed 
in  Appendix  O  regardless  of  their  Operations  Flight  classification. 

The  results  of  the  primary  line  evaluation  are  presented  in  the  next  series  of 
tables.  Each  of  the  161  lines  evaluated  is  contained  in  the  tables  followed  by  a 
representative  example.  Based  upon  the  first  step  the  line  failed,  the  line  ID  number  was 
placed  within  one  of  six  tables  established  for  that  specific  evaluation  step.  Each  of  the 
tables  created  corresponds  to  a  particular  step  within  the  primary  evaluation  and  each 
table  provides  a  list  of  the  line  numbers  that  did  not  pass  the  primary  evaluation 
according  to  what  evaluation  step  failed  (due  to  space  constraints,  just  the  ID  number  will 
be  listed).  For  example.  Table  20,  Lines  That  Did  Not  Contain  a  Defined  Unit  of 
Measure,  lists  the  seven  line  ID  numbers  of  lines  that  did  not  have  a  defined  measure  of 
unit  (question  #1)  along  with  the  Operations  Flight  Function  and  the  appendix  where  the 
line  is  located  in  this  research.  All  lines  passed  the  sensor  question.  Table  23,  Lines  That 
Did  Not  Contain  a  Frequency,  lists  all  of  the  line  ID  numbers  of  lines  not  containing  a 
frequency  (question  #3).  Following  each  table  within  the  series  of  tables,  a  line  that 
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failed  the  specific  step  is  presented  as  an  example,  along  with  the  primary  evaluation,  and 
discussed  in  detail. 


Table  20.  Lines  That  Did  Not  Contain  a  Defined  Unit  of  Measure 


1  ID# 

Operations  Flight  Function 

1  Appendix 

16 

1.1 

1 

70 

1.6 

1 

71 

1.6 

1 

58 

1.7 

j 

59 

1.7 

j 

95 

1.7 

j 

160 

1.7 

J 

Tables  21  and  22  contain  an  example  of  a  line  that  failed  the  first  step  of  the 
primary  evaluation  and  its  primary  evaluation  table.  The  Operations  Flight  Function  and 
the  Gauge  Cluster  classifications  were  removed  from  the  table  because  they  are  not 
needed  during  the  primary  evaluations.  There  is  no  defined  unit  of  measure  for  the 
threshold,  and  the  surveillance.  The  “maintain  a  professional  appearance”  can  be  found 
in  the  objective,  but  professional  appearance  is  not  something  that  has  a  defined  unit  of 
measure  (i.e.,  time,  cost).  This  example  in  Table  4.2  is  typical  of  the  other  six  lines  that 
failed  the  defined  unit  of  measure  step  of  the  primary  evaluation. 


Table  21.  Example  of  a  Line  without  a  Defined  Measure  of  Unii 


m 

Objective 

ElBSM 

Surveillance 

160 

The  service  provider  will  be  expected  to  maintain  a  professional  appearance 
of  all  sites,  roads,  airfield  pavements,  parking  areas,  etc.  Any  deviation  from 
this  professional  level  of  service  (e.g.,  complaints  of  pot  holes,  low  areas 
holding  water  on  the  flightline,  spalls  on  the  airfield,  accumulation  of  debris 
on  streets/pavements,  higher  counts  than  threshold  limits  for  mosquito 
population,  etc.)  will  be  counted  as  a  data  point  in  this  metric.  An  incident  is 
the  first  time  that  an  event  is  brought  to  the  attention  of  the  service  provider. 

A  complaint  is  dissatisfaction  with  the  fix  or  a  lack  of  response  to  the 
incident.  This  metric  will  have  two  columns;  one  for  an  incident  and  one  for 

complaints. 

Monthly 
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Table  22.  Primary  Evaluation  of  Line  #160 


Criteria 

Yes/No 

Justification 

Defined  unit  of  measure? 

No 

There  is  no  specified  unit  of  measure 

Sensor? 

Yes 

Records 

Frequency? 

Yes 

Monthly 

Understandable? 

(Not  difficult  to  understand) 

No 

Due  to  lack  of  specified  unit  of  measure 

Quantifiable? 

(Reduced  personal  influence  or  iudgment) 

No 

Professional  appearance  cannot  be  quantified 

High  Impact? 

(Affect  Quality  of  Life,  Mission,  or 
Customer) 

Yes 

Has  significant  impact  upon  the  customers  of  the 
civil  engineering,  quality  of  life,  and  successful 
completion  of  the  mission 

Table  23  lists  all  the  line  ID  numbers  that  failed  the  third  step  of  the  primary 
evaluation  that  identifies  a  frequency.  There  were  a  total  of  3 1  lines  that  did  not  contain  a 
frequency.  Table  24  contains  an  example  of  a  line  that  did  not  contain  a  frequency  and 
Table  25  is  the  primary  evaluation  of  the  line  example  contained  in  Table  26. 


Table  23.  Lines  without  a  Frequency 


ID# 

Operations  Flight 
Function 

Appendix 

ID# 

Operations  Flight 
Function 

Appendix 

42 

1.1 

D 

50 

1.5 

H 

74 

1.1 

D 

51 

1.5 

H 

75 

1.1 

D 

52 

1.5 

H 

77 

1.1 

D 

53 

1.5 

H 

78 

1.1 

D 

54 

1.5 

H 

87 

1.1 

D 

55 

1.5 

H 

89 

1.1 

D 

88 

1.5 

H 

41 

1.2 

E 

90 

1.5 

H 

80 

1.2 

E 

99 

1.5 

H 

81 

1.2 

E 

118 

1.5 

H 

83 

1.2 

E 

82 

1.6 

I 

98 

1.2 

E 

116 

1.6 

I 

17 

1.4 

G 

117 

1.10 

L 

47 

1.5 

H 

27 

1.12 

M 

48 

1.5 

H 

102 

1.13 

N 

49 

1.5 

H 
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Table  24.  Example  of  Line  without  a  Frequency 


m 

Objective 

Threshold 

Surveillance 

11 

On-site 
response  to 
urgent  work 
orders  within  1 
workday 

95%  of  the  time  and  within  2  work  days  100%  of  the  time.  If 
required  materials  are  not  on-hand,  order  required  materials 
within  7  calendar  days,  100%  of  the  time.  Mitigate  condition  to 
routine  status  or  better  within  7  calendar  days  after  required 
materials  are  available  1 00%  of  the  time. 

Customer 

complaint. 

Table  25.  Primary  Evaluation  of  Dne  #77 


Criteria 

Yes/No 

Justiflcation 

Defined  unit  of  measure? 

Yes 

Response  time  to  urgent  work  orders 

Sensor? 

Yes 

Records  are  maintained 

Frequency? 

No 

No  established  frequency 

Understandable? 

(Not  difficult  to  understand) 

Yes 

Measures  the  time  to  respond  to  an  urgent  work  order 

Quantifiable? 

(Reduced  personal  influence  or 
judgment) 

Yes 

Response  times  are  recorded 

High  Impact? 

(Affect  Quality  of  Life, 

Mission,  or  Customer) 

Yes 

Has  significant  impact  upon  the  customers  of  the  ciyil 
engineering,  quality  of  life,  and  successful  completion  of 
the  mission 

For  a  metric  to  be  well  developed,  it  must  contain  a  frequency.  In  Table  24, 
Example  of  Line  without  a  Frequency,  there  is  not  an  established  frequency  to  measure 
“the  95%  of  the  time”.  There  is  no  specified  period  of  time  to  measure  the  95% 
(monthly,  weekly,  or  yearly).  This  is  typical  of  all  of  the  lines  failing  to  have  a 
frequency.  In  some  cases  the  frequency  may  have  been  identified  in  the  PWS,  but  it  was 
not  contained  in  the  line,  therefore,  the  line  failed  step  3  of  the  primary  evaluation. 

Table  26  lists  all  the  line  ID  numbers  that  failed  the  fourth  step  of  the  primary 
evaluation  that  identifies  if  the  line  is  understandable.  There  were  a  total  of  61  lines  that 
were  found  to  be  not  understandable.  Table  27  contains  an  example  of  a  line  that  was 
not  understandable  and  Table  28  is  the  primary  evaluation  of  the  line  example  contained 
in  Table  27. 
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Table  26.  Lines  not  Understandable 


ID# 

Operations  Flight 
Function 

Appendix 

ID# 

Operations  Flight 
Function 

Appendix 

9 

1.1 

D 

68 

1.2 

E 

32 

1.1 

D 

69 

1.2 

E 

61 

1.1 

D 

79 

1.2 

E 

62 

1.1 

D 

121 

1.2 

E 

63 

1.1 

D 

122 

1.2 

E 

64 

1.1 

D 

123 

1.2 

E 

73 

1.1 

D 

124 

1.2 

E 

94 

1.1 

D 

126 

1.2 

E 

109 

1.1 

D 

127 

1.2 

E 

110 

1.1 

D 

10 

1.5 

H 

111 

1.1 

D 

11 

1.5 

H 

113 

1.1 

D 

12 

1.5 

H 

115 

1.1 

D 

13 

1.5 

H 

135 

1.1 

D 

14 

1.5 

H 

137 

1.1 

D 

15 

1.5 

H 

138 

1.1 

D 

100 

1.5 

H 

101 

1.5 

H 

84 

1.6 

I 

112 

1.5 

H 

104 

1.6 

I 

119 

1.5 

H 

105 

1.6 

I 

120 

1.5 

H 

106 

1.6 

I 

125 

1.5 

H 

107 

1.6 

I 

130 

1.5 

H 

108 

1.6 

I 

131 

1.5 

H 

114 

1.6 

I 

132 

1.5 

H 

160 

1.7 

J 

133 

1.5 

H 

129 

1.8 

K 

134 

1.5 

H 

153 

1.10 

L 

139 

1.6 

I 

25 

1.12 

M 

43 

1.6 

I 

128 

1.12 

M 

44 

1.6 

I 

103 

1.13 

N 

46 

1.6 

I 

57 

1.6 

I 

28 

1.8 

K 

Table  27.  Example  of  Line  Difficult  to  1 

Understand 

m 

Objective 

Threshold 

Surveillance 

69 

Work  Order  Execution:  Complete 
approved  work  orders  (between  50  to 
250  man-hours)  within  10%  of 
approved  cost. 

Timely  resolution  95%  of 
the  time,  measured  on  a 
monthly  interval. 

Periodic  monitoring  of  20-40%  of 
approved  work  orders,  customer 
complaints 

Table  28.  Primary  Evaluation  of  Line  #69 


Criteria 

Yes/No 

Justification 

Defined  unit  of  measure? 

Yes 

Time 

Sensor? 

Yes 

Records  are  maintained 

Frequency? 

Yes 

Monthly 

Understandable? 

(Not  difficult  to  understand) 

No 

There  is  a  mismatch  between  the  Objective 
(cost)  and  the  threshold  (time) 

Quantifiable? 

(Reduced  personal  influence  or  iudgment) 

No 

Due  to  the  mismatch 

High  Impact? 

(Affect  Quality  of  Life,  Mission,  or 
Customer) 

Yes 

Has  significant  impact  upon  the  customers  of 
the  civil  engineering,  quality  of  life,  and 
successful  completion  of  the  mission 

The  line  found  in  Table  27  provides  an  example  of  a  line  that  is  not 
understandable.  The  objeetive  measures  cost  and  the  threshold  identifies  timely 
resolution.  There  is  a  mismatch  between  the  threshold  and  the  objective.  Table  29  and 
Table  30  offer  another  example  of  a  metric  that  was  difficult  to  understand. 


Table  29.  Additional  Example  of  Line  Difficult  to  Understand 


ID 

Objective 

Threshold 

Surveillance 

15 

Operate,  maintain,  and  repair 
mechanical  systems  to  ensure 
temperatures/  humidity  to  the  mission 
critical  centers  are  within  mission 
equipment  parameters,  (^mission 
critical  item 

System  mission  impacting 
failure  time  will  not 
exceed  5.3  minutes 
(99.999%)  annually. 

Monthly  periodic  review  of 
operations  and  repair  actions 
(RWP  records,  consumption  of 
parts  relative  to  RWP  reports, 
and  outage  reports) 

Table  30.  Primary  Evaluation  of  Line  #15 


Criteria 

Yes/No 

Justification 

Defined  unit  of  measure? 

Yes 

Time 

Sensor? 

Yes 

Records 

Frequency? 

Yes 

Monthly 

Understandable?  (Not  difficult  to  understand) 

No 

Due  to  the  objective  being  too  broad 

Quantifiable?(Reduced  personal  influence  or 
judgment) 

No 

Because  the  objective  of  ensuring 
humidity/temp  parameters  cannot  be 
evaluated  by  time 

High  Impact? 

(Affect  Quality  of  Life,  Mission,  or  Customer) 

Yes 

Has  significant  impact  upon  the 
customers  of  the  civil  engineering, 
quality  of  life,  and  successful 
completion  of  the  mission 
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The  line  found  in  Table  29  was  rejected  because  the  objective  was  too  broad. 


There  are  too  many  parameters  in  the  objective  to  be  accurately  measured  by  the 
surveillance  found  in  Table  29.  By  being  so  broad,  the  objective  is  difficult  to  understand 
what  is  being  measured. 

Table  31  lists  all  the  line  ID  numbers  that  failed  the  fifth  step  of  the  primary 
evaluation  that  identifies  if  the  line  is  quantifiable.  There  were  a  total  of  1 1  lines  that 
were  found  to  be  not  quantifiable.  Table  32  contains  an  example  of  a  line  that  was  not 
quantifiable  and  Table  33  is  the  primary  evaluation  of  the  line  example  contained  in 
Table  32. 


Table  31.  Lines  not  Quantifiable 


ID# 

Operations  Flight 
Function 

Appendix 

ID# 

Operations  Flight 
Function 

Appendix 

66 

1.1 

E 

56 

1.5 

H 

1 

1.2 

F 

60 

1.7 

J 

2 

1.2 

F 

146 

1.7 

J 

4 

1.2 

F 

24 

1.12 

M 

18 

1.2 

F 

26 

1.12 

M 

29 

1.2 

F 

Table  32.  Example  of  Line  not  Quantifiable 


EH 

Objective 

Threshold 

Surveillance 

1 

Provide  Production  Control  that  is 
Professional  and  Courteous  at  all  times. 

0  Defects.  Lot  is  number  of 
calls  received  or  verbal  requests 
taken  monthly. 

Customer  Complaint 

Table  33.  Primary  Evaluation  of  Line  #1 


Criteria 

Yes/No 

Justiflcation 

Defined  unit  of  measure? 

Yes 

Professional  and  courteous  personnel 

Sensor? 

Yes 

Customer  records  are  maintained 

Frequency? 

Yes 

Monthly 

Understandable? 

(Not  difficult  to  understand) 

Yes 

Professional  and  courteous  personnel 

Quantifiable? 

(Reduced  personal  influence  or  judgment) 

No 

Personal  judgment  is  not  reduced. 
Professionalism  and  courtesy  are  subjective 
to  customers’  perception 

High  Impact? 

(Affect  Quality  of  Life,  Mission,  or  Customer) 

Yes 

Affects  customers 
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Table  31  provides  an  example  of  a  created  line  that  is  not  quantifiable  because  of 
the  amount  of  personal  judgment  involved  with  classifying  professional  and  courtesy  and 
is  typical  of  the  remaining  10  lines  found  in  Table  30.  The  customer  defines  professional 
and  courteous  based  upon  the  situation.  If  the  customer  perceives  an  employee  is  not 
professional  or  courteous  under  some  circumstance  during  the  interaction  process,  the 
customer  will  not  rate  the  employee  as  professional  and  courteous,  even  if  the  employee 
felt  as  if  the  situation  was  handled  properly.  Because  of  this,  the  professional  and 
courteous  aspect  of  the  work  cannot  be  quantified  because  it  changes  from  customer  to 
customer. 

Table  34  lists  all  the  line  ID  numbers  that  failed  the  sixth  and  final  step  of  the 
primary  evaluation  that  identifies  if  the  line  is  high  impact.  There  were  a  total  of  1 8  lines 
that  were  found  to  be  not  high  impact.  Table  35  contains  an  example  of  a  line  that  was 
not  high  impact  and  Table  36  is  the  primary  evaluation  of  the  line  example  contained  in 
Table  35. 


Table  34.  Lines  not  Classified  as  High  Impact 


ID# 

Operations  Flight 
Function 

Appendix 

ID# 

Operations  Flight 
Function 

Appendix 

8 

1.2 

E 

93 

1.5 

H 

19 

1.2 

E 

85 

1.6 

I 

20 

1.2 

E 

86 

1.6 

I 

37 

1.2 

E 

147 

1.7 

J 

38 

1.2 

E 

148 

1.7 

J 

39 

1.2 

E 

149 

1.7 

J 

40 

1.2 

E 

21 

1.10 

L 

96 

1.2 

E 

22 

1.10 

L 

97 

1.2 

E 

92 

1.5 

H 

Table  35.  Example  of  Line  not  Classified  as  High  Impact 


m 

Objective 

Threshold 

Surveillance 

21 

Ensure  sufficient  number  of  2  -way  radio 
batteries  are  adequately  charged  to  keep  all 
CE  radios  operational  at  all  times 

0  Defects  Lot  is  number  of  radios 
in  CE  during  the  month. 

Checklist 
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Table  36.  Primary  Evaluation  of  Line  #21 


Criteria 

Yes/No 

Justification 

Defined  unit  of  measure? 

Yes 

Number  of  batteries 

Sensor? 

Yes 

Records 

Frequency? 

Yes 

Monthly 

Understandable? 

(Not  difficult  to  understand) 

Yes 

Ensures  batteries  are  adequately  charged 

Quantifiable? 

(Reduced  personal  influence  or  judgment) 

Yes 

Based  upon  the  number  of  radios  not  used 
due  to  insufficient  batteries 

High  Impact? 

(Affect  Quality  of  Life,  Mission,  or 

Customer) 

No 

This  is  an  administrative  issue  that  should  not 
impact  quality  of  life,  customers,  or  mission 

Table  35  identifies  a  line  that  is  not  classified  as  high  impact.  To  ensure  2-way 
radios  have  charged  batteries  is  an  aspect  that  is  internal  to  the  service  provider.  It  is 
agreed  that  if  batteries  are  not  charged,  customers  and  work  to  be  accomplished  will 
suffer,  but  the  line  evaluates  an  issue  that  is  administrative.  Therefore,  it  is  classified  as 
not  being  high  impact  and  is  representative  of  the  other  17  lines  listed  in  Table  34. 

Some  of  the  lines  passed  the  primary  line  evaluation  but  were  not  evaluated 
further  because  the  Civil  Engineer  unit  does  not  or  has  not  begun  to  use  the  standards  and 
metrics.  Of  the  remaining  33  lines  that  passed  the  primary  evaluation,  eighteen  lines  (line 
ID  Numbers:  76,  91,  140,  141,  142,  143,  144,  145,  150,  151,  152,  154,  155,  156,  157, 

158,  159,  and  161)  were  not  further  evaluated  because  the  bases  had  not  implemented  the 
service  provider  at  the  time  of  this  research.  They  were  eliminated  from  the  secondary 
evaluation  because  the  secondary  evaluation  involved  collection  of  data  and  storage  of 
data,  which  does  not  currently  take  place  at  the  bases  from  where  the  1 8  lines  were 
collected.  Also,  eight  lines  (line  ID  Numbers:  3,  23,  30,  31,  33,  34,  35,  36)  were  not 
evaluated  because  the  base  where  the  lines  were  obtained  from  uses  command  metrics  for 
evaluation,  not  the  QASP.  The  26  lines  not  evaluated  further  are  listed  in  Appendix  O 
but  there  will  not  be  a  secondary  evaluation  table  with  them.  There  is  a  discussion  in  the 


78 


results  section  regarding  bases  not  using  QASP  documents.  Within  the  26  lines  not 
evaluated  by  the  secondary  evaluation,  the  majority  of  the  Operations  Flight  Functions 
(65%)  were  found  in  the  1.1,  Operate,  Maintain,  and  Repair,  and  1.7,  Quality  Standards. 
Also,  a  majority  of  the  gauge  clusters  (65%)  were  categorized  as  quality  or 
time/schedule.  The  seven  lines  that  passed  the  primary  evaluation  will  be  discussed  in 
the  Secondary  Line  Evaluation  section.  The  following  section  outlines  the  steps  taken  to 
further  evaluate  the  seven  lines  that  passed  the  primary  line  e\aluation. 

Secondary  Line  Evaluation 

The  puipose  of  the  secondary  evaluation  was  to  provide  a  quantitative  evaluation 
of  the  standard  and  metric  lines  that  passed  the  primary  evaluation.  The  secondary 
evaluation  consisted  of  specific  information  obtained  from  the  respective  bases.  The 
secondary  line  evaluation  was  similar  to  the  primary  with  a  list  of  five  steps  to  evaluate 
each  of  the  seven  lines.  Because  the  researcher  could  not  adequately  evaluate  the  steps, 
the  bases  from  which  the  standards  and  metrics  came  were  contacted  to  further  assist  in 
the  evaluations.  The  five  steps  used  in  the  secondary  evaluation  were:  1)  Is  the  objective 
measured?  2)  is  the  collected  information  stored?  3)  How  long  is  the  collected 
information  stored?  4)  Is  the  line  cost  effective?;  and  5)  Is  the  line  proven  to  show 
results?  The  five  questions  of  the  secondary  evaluation  were  applied  to  each  of  the 
seven  lines  that  passed  the  primary  evaluation.  The  entire  list  of  lines  that  underwent  the 
secondary  evaluation  can  be  found  in  Appendix  O  along  with  their  respective  secondary 
evaluation  table.  Appendix  O  also  contains  the  26  lines  that  passed  the  primary 


79 


evaluation  but  are  not  included  in  the  secondary  evaluation  steps.  Table  37  lists  the  line 
ID  numbers  of  the  lines  that  passed  the  primary  evaluation. 


Table  37.  Lines  Used  in  the  Secondary  Line  Evaluations 


1  ID# 

Operations  Flight  Function 

1  Appendix 

7 

1.1 

D 

67 

1.1 

D 

5 

1.3 

F 

65 

1.3 

F 

45 

1.6 

I 

72 

1.6 

I 

6 

1.7 

J 

Six  of  the  seven  lines  involved  in  the  secondary  evaluation  passed  the  five  step 
secondary  evaluation;  one  is  displayed  in  Table  38,  and  its  corresponding  secondary 
evaluation  in  Table  39  (all  information  used  to  complete  the  secondary  evaluation  tables 
was  obtained  from  phone  interviews  with  the  respective  base  personnel). 


Table  38.  Example  Line  Passing  the  Secondary  Evaluation 


ID 

Objective 

Threshold 

Surveillance 

5 

Emergency  Work  Requests:  30  minutes  (duty 
hours)/ 1  hour  (non-duty  hours);  completed  (safed) 
in  24  hours. 

1 00%  of  the  time 

Records  Review  or 
Customer  Contact,  at 
least  1/week 

Table  39.  Secondary  Evaluation  of  Line  #5 


Criteria 

Yes/No 

Justification 

Is  objective  measured? 

Yes 

By  contractors  telling  QAE  he  will  not  make  the 
required  time 

Where  is  collected  information  stored? 
(Accessible  to  those  that  need 
information) 

IWIMS 

How  long  is  the  collected  information 
stored? 

1  year 

Cost  Effective? 

(Value  of  obtained  information 
outweighs  cost  of  seeking  information) 

Yes 

Very  little  cost  involved 

Proven? 

(Has  shown  demonstrated  results) 

Yes 

If  the  work  request  is  not  completed  in  specified 
time,  the  QAE  questions  the  service  provider  and 
initiates  closure  as  close  to  the  specified  time  as 
possible 
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Each  of  the  six  lines  passing  the  secondary  evaluation  is  similar  to  the  example 
line  given  in  Table  38  because  they  all  involved  a  service  to  the  base  (work  order  and 
base  services).  Each  standard  and  metric  is  measured  or  tracked.  The  only  difference 
between  the  standards  and  metrics  is  where  the  information  is  stored.  Some  have  IWIMS 
for  data  storage  and  others  use  records  or  logs  for  the  information.  The  data  for  the  six 
standards  and  metrics  is  kept  for  a  year,  which  will  be  further  discussed  in  the  Results 
section  The  standards  and  metrics  are  very  cost  effective  because  the  data  are  stored  in 
systems  that  have  already  been  established  (IWIMS  or  logs)  and  a  QAE  ensures  the  work 
is  accomplished.  The  standards  and  metrics  are  proven  because  the  service  provider 
understands  that  if  the  work  is  not  completed  by  the  timeline,  customer  complaints  will 
begin  and  the  service  provider  may  have  to  add  the  work  to  what  is  already  been 
scheduled  for  the  next  week  or  month,  which  may  result  in  a  backlog  of  work  or  having 
personnel  work  extra  hours.  Therefore,  by  knowing  the  information  is  collected,  it  is  in 
the  best  interest  of  the  service  provider  to  complete  the  work  in  the  required  time  as 
stated  in  the  lines  (or  PWS).  Table  40  and  41  provide  an  example  of  the  line  that  did  not 
pass  the  secondary  evaluation. 


Table  40.  Example  Line  not  passing  the  Secondary  Evaluation 


ng 

Objective 

Threshold 

Surveillance 

72 

Refuse  Collection:  Collect  and  dispose  of  waste 
(After  contractor  assumption  of  tasks  Oct  2001). 
Refuse  is  picked  up  and  disposed  of  1  AW  with 
SOW  standards  and  schedules. 

Completed  on  time.  Pick  up 
schedule  deviation  is  less  than  5%, 
measured  on  a  monthly  basis. 

Customer 

complaints 
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Table  41.  Secondary  Evaluation  of  Line  #72 


Criteria 

Yes/No 

Justification 

Is  objective  measured? 

No 

Involves  review  of  schedules  if  a  customer 
complaint  is  validated 

Where  is  collected  information  stored? 
(Accessible  to  those  that  need 
information) 

No 

How  long  is  the  collected  information 
stored? 

NA  due  to  the  information  not  being  stored 

Cost  Effective? 

(Value  of  obtained  information 
outweighs  cost  of  seeking  information) 

No 

It  is  not  known  if  the  line  is  cost  effective 

Proven? 

(Has  shown  demonstrated  results) 

No 

Service  provider  just  started  in  Oct  of  2001 

The  line  found  in  Table  40  did  not  pass  any  of  the  five  steps  within  the  seeondary 
evaluation.  A  new  serviee  provider  was  just  implemented  (3  months  prior  to  this 
researeh)  and  the  information  was  not  eolleeted  up  to  the  time  of  the  line  evaluation.  It 
was  not  known  at  the  time  of  this  research  if  the  line  would  be  cost  effective  or  proven. 

Six  lines  from  the  original  161  lines  passed  the  primary  and  secondary 
evaluations,  which  indicates  they  were  properly  designed,  and  is  supported  by  the 
quantitative  evaluation.  Five  of  the  lines  were  work  order  and  DSW  response  times 
(Operations  Flight  Functions  1.1  Operate,  Maintain,  and  Repair;  and  1.3  Emergency 
Response)  identified  as  time/schedule  gauge  clusters  and  one  was  a  pest  removal  service 
(Operations  Flight  Function  1 .6,  Base  Support)  identified  as  an  Operational  Service  Level 
gauge  cluster.  This  leaves  many  of  the  Operations  Flight  Functions  without  properly 
designed  metrics  and  standards  (in  appropriate  gauge  clusters)  to  sufficiently  evaluate 
them. 
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Evaluation  of  Operations  Flight  PWS  Templates  and  Operations  Flight  Metrics 

The  Air  Force  Civil  Engineering  community  has  many  guidelines  to  help 
streamline  the  competitive  sourcing  process.  One  of  the  guidelines  is  an  Operations 
Flight  Template  found  on  the  AFCESA  website.  This  research  reviewed  the  contents  of 
the  template  to  determine  which  Operations  Flight  critical  areas  are  currently  evaluated 
by  the  templates  and  if  standards  and  metrics  could  be  properly  designed  from  the 
templates.  AFCESA  also  has  a  separate  list  of  metrics  that  was  introduced  in  Chapter  3. 
The  templates  and  the  metrics  were  evaluated  to  determine  if  sufficient  guidance  exists  to 
help  bases  undergoing  the  competitive  sourcing  process  to  develop  standards  and  metrics. 

The  Operations  Flight  Function  and  gauge  cluster  classifications  for  the  metrics 
found  in  the  templates  and  AFCESA  Operations  Metrics  can  be  found  in  Appendices  P 
and  Q,  respectively.  Each  metric  was  extracted  and  classified  according  to  its  Operations 
Flight  Function  and  gauge  cluster.  Then,  the  primary  evaluation  was  applied  to  each  of 
the  metrics.  Only  the  primary  evaluation  was  used  because  the  metrics  are  representative 
samples.  The  primary  evaluation  for  each  of  the  metrics  found  within  the  AFCESA 
templates  and  Operations  Metrics  can  be  found  in  Appendix  R  and  S,  respectively. 

Evaluation  of  PWS  &  QASP  Templates 

AFCESA  provides  many  templates  for  civil  engineering  on  its  website.  The 
particular  template  chosen  for  this  research  was  specific  to  the  Operations  Flight.  Each 
of  the  standards  and  metrics  found  in  the  service  delivery  summary  of  the  PWS  template 
were  individually  evaluated  in  the  same  manner  as  the  performance  standard  and  metric 
lines.  The  standards  and  metrics  found  on  the  templates  were  reviewed  to  determine  if 


83 


the  templates  provided  are  adequate  in  supporting  the  mission  of  the  Operations  Flight.  It 
is  stated  on  the  template  that  the  individual  bases  need  to  tailor  the  metrics  to  their 
specific  needs.  Table  42  lists  the  nine  standards  and  metrics  found  in  the  Operations 
Flight  PWS  and  QASP  templates. 


Table  42.  PWS  &  QASP  Template  Standards  and  Metrics 


ID 

SOW 

para. 

Performance  Objective 

Performance 

Threshold 

T-1 

Treat  customers  politely,  cheerfully  and  promptly 

1.1.1 

Customer  service 
rating  of  at  least  4.0 
on  a  5.0  scale 

T-2 

Respond  to  and  complete  emergency,  urgent,  and 
routine  service  calls 

1.1. 1.1 

and 

1.1.3. 

95%  of  service  calls 
are  responded  to  and 
completed  timely 

T-3 

Maintain,  repair,  construct,  and  operate  the  supporting 
infrastructure  ensuring  cost  effective  and  reliable 
support 

1.2. 

100%  of  time 

T-4 

Provide  economical  maintenance,  repair,  construction, 
installation,  operation,  and  service  functions  for  real 
property,  Real  Property  Installed  Equipment  (RPIE), 
and  designated  Equipment  Authorized  Inventory  Data 
(EAID) 

1.3. 

95%  of  scheduled 
inspections  and/or 
work  completed  on 
time 

Operate  and  maintain  steam  heat,  hot  water,  chilled 
water,  water  treatment,  and  wastewater  treatment  plants 

1.4. 

1 00%  of  time 

T-6 

Provide  facility  and  base  services  protecting  public 
health  and  base  property,  which  includes  custodial 
services,  refuse  collection  and  disposal,  locksmith 
services,  pest  and  animal  control,  hazardous  waste 
collection  and  disposal,  snow  and  ice  control,  grounds 
maintenance  and  landscaping.  Provides  services  in 
accordance  with  submitted  plans 

1.5. 

Schedules  are  met 

95%  of  time 

T-7 

Provide  an  effective  design  program  and  construction 
management  to  ensure  projects  are  designed  and 
construction  work  are  completed  timely  and  within 
budget. 

1.6. 

100%  of  time 

T-8 

Develop  the  CE  Financial  Plan 

1.7. 

100%  of  time 

T-9 

Respond  to  contingencies  and  natural  disasters  during 
normal  and  after  duty  hours  within  required  time  limits 

1.8. 

100%  of  time 

Once  the  metrics  were  classified  according  to  Operations  Flight  Function  and 
gauge  cluster,  the  classifications  were  sorted  and  summed  in  Table  43,  Classification  of 
AFCESA  Template  Metrics.  The  results  of  the  AFCESA  template  evaluations  were 
significant  in  that  many  of  the  metrics  are  improperly  designed  and  many  critical  areas 
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are  not  evaluated.  Table  43,  Classification  of  AFCES A  Template  Metrics,  identifies  the 
Operation  Flight  Function  and  Gauge  Clusters  for  the  nine  metrics  found  in  the 
Operations  Flight  Template. 


Table  43.  Classification  of  AFCESA  Template  Metrics 


Operation  Flight 
Functions 

Gauge  Clusters 

HR 

T/S 

OSL 

Quality 

F/B 

cs 

WPD 

Pro 

Total  number  of  metrics 
found  within  Operations 
Flight  Function 
Classification; 

1.1,  Operate, 
maintain,  and  repair 

0 

0 

2 

0 

0 

0 

0 

0 

2 

0 

0 

0 

0 

0 

1 

0 

0 

1 

1.3,  Emergency 
response 

0 

2 

0 

0 

0 

0 

0 

0 

2 

1.4,  Compliance 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1.5,  Reliable  utilities 

0 

0 

n 

0 

0 

0 

0 

0 

1 

1.6,  Base  support 

0 

0 

0 

0 

0 

0 

D 

0 

1 

1.7,  Quality 
standards 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1.8,  Self  help 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1.9,  Future  plans 

0 

1 

0 

0 

0 

0 

0 

0 

1 

1.10,  Allocate 
Resources 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1.11,  Provide  costs 
to  customers 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

1 

1.13,  Logistics 
support 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1.14,  Facility 
manager  program 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Total  number  of 
metrics  found  within 
gauge  cluster 
classification: 

0 

3 

3 

0 

1 

1 

■ 

0 

9 

From  the  classification,  4  (44%)  are  classified  as  Operations  Flight  Functions  1.1, 
Operate,  Maintain,  and  Repair,  and  1.3,  Emergency  response.  The  majority  (67%)  of  the 
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gauge  cluster  classifications  are  found  in  operational  service  level  and  time/schedule. 

Half  of  the  14  Operations  Flight  Functions  are  not  evaluated  by  the  metrics  found  in  the 
template. 

The  application  of  the  primary  line  evaluation  was  the  next  step.  Each  of  the 
metrics,  which  can  be  located  in  Appendix  P  with  their  corresponding  primary 
evaluation,  failed  the  sensor  and  frequency  portion  of  the  evaluation.  Two  of  the  nine 
can  be  corrected  by  establishing  an  appropriate  sensor  and  frequency.  The  remaining 
seven  require  changes  in  their  design  because  they  are  very  broad  and  encompass  many 
aspects  of  a  process. 

The  sample  standards  and  metrics  found  within  the  service  delivery  summary  of 
the  templates  are  very  broad  and  should  not  be  used  as  they  are  written.  They  should  be 
used  as  a  guide  to  identify  the  areas  that  need  to  be  evaluated.  It  must  be  noted  that  the 
template  does  state  that  the  individual  bases  must  write  its  own  unique  requirements  and 
quantities.  Given  that,  the  templates  are  still  poor  in  that  the  standards  do  not  specify 
what  critical  areas  should  be  evaluated.  For  example,  T-3  found  in  Table  42  identifies 
“Maintain,  repair,  construct,  and  operate  the  supporting  infrastructure”.  T-3  is  an 
encompassing  metric;  it  does  not  specify  what  is  the  critical  aspect;  the  operation, 
maintenance,  or  repair.  Base  leadership  decides  the  importance  of  each  aspect,  and  the 
template  identifies  them  all  as  a  suggestion  to  what  may  be  important.  The  base 
leadership  at  each  base  must  decide  what  aspect  to  focus  the  metrics  on.  Also,  because 
the  standards  are  very  broad,  much  confusion  is  introduced  as  to  what  should  the  standard 
be  collecting  for  data.  Many  of  the  lines  evaluated  in  this  research  were  very  similar  to 
the  metrics  found  in  the  template  shown  in  Table  42.  Many  of  those  did  not  pass  the 
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primary  evaluation  because  they  were  not  understandable  or  quantifiable.  T-3  has  an 
objective  to  “maintain,  repair,  operate,  and  construct”  and  the  performance  threshold 
states  these  must  be  met  100%  of  the  time.  This  creates  a  confusing  standard  and  metric 
and  unfortunately  is  commonly  found  on  the  PWS  and  QASP  templates.  It  is  not  clear 
as  to  what  is  meant  by  repair  100%  of  the  time  or  100%  construct  (for  example, 
economical,  quality,  or  timely).  This  research  used  some  of  the  information  found  within 
the  templates  to  recommend  metrics  that  may  be  used  Air  Force  wide  in  competitive 
sourcing  efforts.  Those  metrics  are  outlined  in  the  Proposed  Metrics  section. 

Evaluation  of  Operations  Fli2ht  Metrics 

The  AFCESA  Operations  Flight  Metrics,  located  in  Appendix  B,  were  also 
evaluated  using  the  same  primary  evaluation  applied  to  the  161  standard  and  metric  lines 
and  the  AFCESA  template  metrics.  The  AFCESA  Operations  Flight  Metrics  were 
individually  classified  according  to  their  Operation  Flight  Function  and  gauge  cluster. 

The  classifications  of  the  metrics  are  sorted  and  summed  in  Table  44,  Classification  of 
AFCESA  Operations  Flight  Metrics.  The  findings  for  the  47  AFCESA  metrics  were 
similar  to  that  of  the  templates  because  only  two  Operations  Flight  Functions  and  only 
three  gauge  clusters  contain  a  majority  of  the  metrics.  Two  Operations  Flight  Functions, 
1.1,  Operate,  Maintain,  and  Repair,  and  1.12,  Time  and  material  accounting,  contained  22 
(47%)  of  the  metrics.  The  time/schedule,  operational  service  level,  and  work  product 
delivered  gauge  clusters  accounted  for  34  (72%)  of  the  metrics.  Many  areas  are  not 
sufficiently  evaluated  by  the  AFCESA  metrics.  Table  44,  Classification  of  AFCESA 
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Operations  Flight  Metrics,  identifies  the  Operation  Flight  Functions  and  gauge  clusters 
for  the  47  metrics. 


Table  44.  Classification  of  AFCESA  Operations  Flight  Metrics 


Operation  Flight 
Functions 

Gauge  Clusters 

HR 

T/S 

OSL 

Quality 

F/B 

CS 

WPD 

Pro 

Total  number  of  metrics 
found  within  Operations 
Flight  Function 
Classification; 

1.1,  Operate, 
maintain,  and  repair 

0 

2 

D 

0 

0 

0 

4 

0 

10 

0 

0 

0 

0 

0 

0 

1 

0 

1 

1.3,  Emergency 
response 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1.4,  Compliance 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1.5,  Reliable  utilities 

0 

0 

3 

1 

0 

0 

2 

0 

6 

1.6,  Base  support 

0 

0 

2 

0 

0 

0 

0 

0 

2 

1.7,  Quality 
standards 

0 

4 

D 

1 

0 

1 

D 

0 

8 

1.8,  Self  help 

0 

0 

0 

0 

D 

0 

D 

0 

2 

1.9,  Future  plans 

0 

0 

0 

0 

0 

0 

D 

0 

1 

1.10,  Allocate 
Resources 

2 

0 

D 

0 

0 

0 

0 

1 

4 

1.11,  Provide  costs 
to  customers 

0 

0 

0 

0 

0 

0 

0 

0 

0 

PHSHH 

0 

4 

0 

1 

1 

0 

2 

4 

12 

1.13,  Logistics 
support 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1.14,  Facility 
manager  program 

0 

0 

1 

0 

0 

0 

0 

0 

1 

Total  number  of 
metrics  found  within 
gauge  cluster 
classification: 

2 

10 

12 

3 

2 

1 

12 

5 

47 

The  primary  line  evaluation  was  applied  to  each  of  the  47  AFCESA  metrics  as  the 
next  step.  Each  of  the  metrics,  which  can  be  located  in  Appendix  Q  with  their 
corresponding  primary  evaluation,  failed  the  sensor  portion  of  the  evaluation;  and  all 
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metrics  but  nine  failed  the  frequency  portion.  Seventeen  of  the  metrics  can  be  corrected 
by  establishing  an  appropriate  sensor  and  frequency. 

Some  of  the  47  metrics  listed  by  AFCESA  tend  to  evaluate  processes  as  if  the 
major  command  or  the  CE  unit  is  the  customer,  and  because  the  161  created  lines  were 
evaluated  from  a  base  personnel  viewpoint,  27  of  the  46  AFCESA  metrics  were  classified 
as  not  high  impact.  For  example,  M-10  from  Appendix  B  measures  RWP  hours  by  zone, 
(some  CE  units  break  work- orders  out  according  to  zones,  others  break  work  out 
according  to  shops),  and  base  personnel  may  not  be  interested  in  this  aspect,  therefore  M- 
10  was  not  used.  Also,  many  of  the  AFCESA  metrics  can  be  classified  as  a 
time/schedule  gauge  cluster.  But  when  evaluating  the  time/schedule  aspect  of  the 
function,  the  metrics  are  not  evaluating  the  quality  of  the  work  (done  right  the  first  time), 
the  cost  effectiveness  of  the  work,  or  the  customer  service  aspect  of  the  work  completed. 
All  of  which  would  be  measured  if  the  customer  were  the  primary  customer,  not  the 
major  command.  In  another  example,  the  numbers  of  self-help  customers  are  tracked 
(M-33).  Neither  the  customer  satisfaction  nor  the  financial  aspect  of  the  self-help  store  is 
evaluated,  just  the  number  of  customers,  which  is  compared  to  the  previous  year.  This  is 
something  the  major  command  and  the  civil  engineering  unit  is  interested  in,  but  has  no 
purpose  for  improving  effectiveness.  If  numbers  for  the  self-help  are  low,  the  civil 
engineering  unit  cannot  do  much  to  improve  the  situation,  except  increase  awareness  (but 
this  metric  does  not  evaluate  awareness). 

Although  the  AFCESA  metrics  and  templates  cannot  be  used  as  is,  they  do 
provide  a  starting  point  for  this  research  effort  to  create  specific  metrics  to  be  used  in 
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future  competitive  sourcing  efforts.  The  proposed  metrics  are  presented  in  the  following 
section. 


Proposed  Metrics 


There  were  four  steps  to  creating  the  proposed  metrics:  create  focused  functions, 
select  focused  functions  that  impact  base  personnel  as  the  customer,  consolidate  functions 
and  gauge  clusters,  and  provide  metrics  on  critical  areas  that  are  formed.  The  first  step  is 
to  divide  the  current  14  Operations  Flight  Functions  (Table  9,  Civil  Engineer  Operations 
Flight  Functions)  into  focused  functions.  The  focused  functions  are  more  specific  in 
identifying  the  critical  areas  and  direct  the  researcher  in  developing  metrics  to  evaluate 
the  critical  areas.  For  example,  the  “operate,  maintain,  and  repair”  Operations  Flight 
Function  (1.1)  is  identified  as  having  three  areas  of  focus:  operate,  repair,  and  maintain, 
and  each  is  listed  separately  in  Table  45,  Focused  Operations  Flight  Functions.  Table  45 
lists  the  21  focused  functions  developed  from  the  14  Operations  Flight  Functions.  This 
step  is  necessary  to  ensure  all  critical  areas  of  the  14  Operations  Flight  Functions  are 
identified  and  to  ensure  the  areas  will  have  metrics  designed  to  evaluate  them. 


Table  45.  Focused  Operations  Flight  Functions 


ID 

Focused  Function 

1 

Operate  systems 

2 

Maintain  systems 

3 

Repair  systems 

4 

Provide  trained  personnel 

5 

Maintain  capability  to  respond  to  emergency  conditions 

6 

Compliance  with  applicable  environmental  laws,  codes  and  directives 

7 

Compliance  with  applicable  fire  laws,  codes  and  directives 

8 

Compliance  with  applicable  safety  laws,  codes  and  directives 

9 

Provide  reliable  utilities 

10 

Provide  cost-effective  utilities 

11 

Provide  base  support  services 

12 

Establish  quality  mechanisms 

13 

Establish  Self  Help  system 

14 

Develop  future  plans 

90 


15 

Update  future  plans 

16 

Allocation  of  resources 

17 

Provide  customers  with  costs  of  work  or  services  performed  on  their  facilities 

18 

Maintain  a  time  accounting  system 

19 

Maintain  a  material  accounting  system 

20 

Proyide  effectiye  logistics  support 

21 

Proyide  a  facility  manager  program 

The  second  step  in  creating  the  proposed  metrics  is  identifying  which  of  the 
focused  functions  impact  base  personnel  as  the  customer  (as  was  done  in  the  primary 
evaluation  of  the  161  lines).  There  were  potentially  three  customer  viewpoints 
considered  for  this  research:  MAJCOM,  CE  unit,  or  base  personnel.  It  was  decided  that 
the  base  personnel  viewpoint  was  the  most  critical  because  civil  engineer  units  are 
service  organizations  serving  the  base  populace  for  a  majority  of  the  time.  Therefore,  the 
determination  that  base  personnel  are  the  customers  focuses  the  standards  and  metrics  to 
be  created  on  the  customer.  It  is  important  that  the  distinction  is  made  because  of 
potential  conflicts  that  may  arise.  For  example,  repairing  a  broken  utility  quickly  with 
additional  personnel  (focused  on  the  customer  from  a  time/schedule  view)  and,  repairing 
it  as  economically  as  possible  with  fewer  personnel  (focused  on  CE  unit  and  MAJCOM 
from  a  finance/budget  view)  may  conflict  because  of  the  additional  personnel  that  may  be 
required  to  repair  the  break  quickly.  The  additional  personnel  would  cost  the  CE  unit  and 
MAJCOM  more,  but  the  repair  may  be  completed  sooner  than  if  the  additional  personnel 
were  not  available.  The  decision  of  which  standards  and  metrics  to  evaluate  for  what 
customer  viewpoint  must  come  from  leadership,  which  complicates  the  issue  of  standard 
and  metric  design  because  of  the  variance  in  which  customers  the  leadership  feels  are 
most  important.  Table  46,  Functions  Important  to  Base  Personnel  as  Customers,  lists  the 
only  the  focused  Operations  Flight  Functions  (from  Table  45)  that  may  apply  to  base 
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personnel  as  customers.  Some  of  the  functions  received  a  new  ID  number  because  four 
of  the  functions  were  removed  from  the  list  that  is  presented  in  Table  45. 


Table  46.  Functions  Important  to  Base  Personnel  as  Customers 


ID 

Focused  Function 

1 

Operate  systems 

2 

Maintain  systems 

3 

Repair  systems 

4 

Provide  trained  personnel 

5 

Maintain  capability  to  respond  to  emergency  conditions 

6 

Compliance  with  applicable  environmental  laws,  codes  and  directives 

7 

Compliance  with  applicable  fire  laws,  codes  and  directives 

8 

Compliance  with  applicable  safety  laws,  codes  and  directives 

9 

Provide  reliable  utilities 

10 

Provide  cost-effective  utilities 

11 

Provide  base  support  services 

12 

Establish  Self  Help  system 

13 

Develop  future  plans 

14 

Update  future  plans 

15 

Provide  customers  with  costs  of  work  or  services  performed  on  their  facilities 

16 

Provide  effective  logistics  support 

17 

Provide  a  facility  manager  program 

Each  of  the  focused  functions  in  Table  46  are  important  to  base  personnel  as 
customers.  Table  47,  Excluded  Functions,  contains  the  functions  removed  from  the 
critical  area  list  because  the  functions  are  internal  to  the  civil  engineering  unit.  The  civil 
engineering  unit  or  MAJCOM  has  greater  interest  in  these  four  critical  areas  than  base 
personnel.  Base  personnel  are  not  directly  affected  by  each  of  the  critical  areas  listed  in 
Table  47.  The  ID  number  is  representative  of  the  ID  number  found  in  Table  45. 


Table  47.  Excluded  Functions 


ID 

Focused  Function 

12 

Establish  quality  mechanisms 

16 

Allocation  of  resources 

18 

Maintain  a  time  accounting  system 

19 

Maintain  a  material  accounting  system 
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The  third  step  of  the  proposed  evaluation  system  was  to  consolidate  all  gauge 
clusters  and  Operations  Flight  Functions  into  a  matrix.  Many  of  the  17  focused  functions 
from  Table  46,  and  nine  gauge  clusters  (Table  48,  Gauge  Clusters)  have  duplicates  or 
they  have  similar  functions  that  can  be  combined.  This  is  done  to  reduce  the  duplication 
of  effort  in  creating  performance  metrics.  The  other  consolidations  involve  combining 
the  Operations  Flight  Functions  with  a  duplicate  gauge  cluster,  combining  duplicate 
gauge  clusters,  and  also  deleting  gauge  clusters  not  used.  Table  49,  Explanation  of 
Consolidated  Gauge  Clusters  and  Focused  Functions,  lists  how  and  why  the  gauge 
clusters  and  focused  functions  were  consolidated. 


Table  48.  Gauge  Clusters 


Gauge  Cluster 

Objective 

Finance/Budget 

Cost  management  and  on-cost  delivery  of 
services 

Customer  Satisfaction 

Critical  attributes  that  generate  satisfaction  with 
services  and  work  products  among  internal 
business  customers 

Work  Product  Delivered 

Quantifying  the  amount  of  service  or  work 
provided  in  a  given  time  period 

Quality 

Objective  and  measurable  aspects  of  quality  of 
services  and  products 

Time/Schedule 

Critical  service,  product,  and  project  time  frames 
and  the  ability  to  deliver  on-time 

Business  Value 

Measures  the  outsourcing  agreement’s  outcome 
attainment  from  the  financial/ shareholder  view, 
external  customer/marketplace  view, 
organizational  learning  and  improvement  view, 
and  internal  process  improvement  view 

Operational  Service  Levels 

Critical  service  tempos,  availability,  and  delivery 
of  work  products 

Human  Resources 

Changes  to  the  skill  inventory  and  internal  job 
satisfaction 

Productivity 

Efficiency  of  the  production  and  delivery  of 
work  products 
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Table  49  identifies  the  consolidation  of  both  gauge  clusters  and  focused  functions, 


listed  numerically  and  in  random  order,  and  also  lists  the  justification  to  the 
consolidation. 


Table  49.  Explanation  of  Consolidated  Gauge  Clusters  and  Focused  Functions 


Consolidation 

The  Business  Value  gauge  cluster  was  deleted  because  it  measures  outcomes  of  the  outsourcing 
agreement.  The  researcher  felt  this  did  not  impact  base  personnel,  but  may  be  more  for  taxpayers. 

Human  Resources  gauge  cluster  was  deleted  because  it  emphasizes  the  civil  engineering  unit  as  the 
customer. 

Operational  Service  Level  gauge  cluster  emphasizes  the  civil  engineering  unit  as  the  customer; 
therefore  it  was  deleted. 

Customer  Satisfaction  gauge  cluster  will  be  met  by  satisfying  time/schedule,  quality,  finance/budget, 
and  work  product  delivered  gauge  clusters.  If  these  four  gauge  clusters  are  met,  the  customer  should 
be  satisfied. 

Response  function  is  listed  under  time/schedule  gauge  cluster  with  categories  for  emergency,  urgent, 
and  routine  work  orders  listed  separately.  Response  is  directly  related  to  time/schedule. 

Trained  personnel  function  is  applied  to  time/schedule,  quality,  work  product  delivered,  and 
finance/budget  gauge  clusters.  All  the  gauge  clusters  require  trained  personnel;  if  one  gauge  cluster 
is  not  performing  well,  it  may  be  an  indication  that  more  training  is  required. 

Safety  compliance,  fire  compliance,  and  environment  compliance  are  included  under  the  quality 
gauge  cluster.  If  aspects  of  safety,  fire,  and  environment  are  met,  some  aspects  of  quality  should  be 
satisfied.  Each  segment  of  conpliance  (fire,  safety,  and  environment)  must  have  its  own  metrics  to 
evaluate  what  is  deemed  important  by  leadership. 

Base  support  function  and  logistics  support  function  are  listed  under  a  new  heading  of  installation 
support  to  simplify  the  creation  of  metrics. 

Cost  information  function  is  listed  under  self-help  function  because  cost  is  directly  related  to 
determining  if  the  project  can  be  done  with  self-help.  Self-help  is  the  only  aspect  that  affects  the 
Operations  Flight.  For  example,  if  the  work-order  became  a  project,  the  project  control  would  shift  to 
the  Engineering  Flight. 

Productivity  gauge  cluster  emphasizes  the  civil  engineering  unit,  but  is  moved  to  time/schedule 
because  productivity  may  be  a  root  cause  of  not  obtaining  the  time/schedule  gauge  cluster. 

Develop  plan  function  and  update  plan  functions  are  moved  under  the  facility  manger  program 
because  they  are  directly  related  in  that  plans  involve  facilities  and  surrounding  areas. 

The  consolidated  list  of  gauge  clusters  and  Operations  Flight  Functions  found  in 
Table  49  are  listed  in  matrix  form  in  Table  50,  Cross  Reference  of  Gauge  Clusters  and 
Specific  Operations  Flight  Functions,  with  the  new  gauge  clusters  listed  vertically  in  the 
table  and  the  new  functions  listed  horizontally.  If  a  gauge  cluster  was  applicable  to  a 
function,  from  a  base  personnel  perspective,  it  was  identified  in  Table  50  with  an  “X”.  A 
metric  was  developed  (by  reviewing  applicable  standards  and  metrics  from  this  research. 
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AFCESA  template,  and  Operations  Flight  metrics)  for  each  “X”  found  within  Table  50 
and  presented  as  a  recommendation  for  use  in  future  competitive  sourcing  efforts.  The 
information  contained  within  the  cells  of  Table  50  is  not  all-inclusive;  they  just  list  the 
changes  made  by  this  research  effort.  For  example,  the  Quality  cell  under  gauge  clusters 
does  not  contain  just  trained  personnel,  fire  compliance,  safety  compliance,  environment 
compliance,  and  customer  satisfaction.  These  five  are  listed  under  Quality  to  suggest 
how  the  original  Operations  Flight  Functions  and  Gauge  Clusters  can  be  combined.  The 
combinations  create  mutually  exclusive  functions  and  gauge  clusters  and  simplify  the 
creation  of  the  proposed  metrics. 

From  Table  50,  an  “X”  marking  indicates  a  function  is  important  to  the  base 
personnel  given  the  specific  gauge  cluster,  as  identified  by  the  researcher.  For  example, 
the  specific  function  1  intersects  with  work  product  delivered  gauge  cluster.  This 
indicates  customers  are  interested  in  the  output  of  the  system.  In  many  cases,  the  output 
level  desired  by  the  base  personnel  is  100%.  For  example,  base  personnel  want  quality 
electricity  (no  brown  outs)  or  clean  water,  100%  of  the  time.  They  typically  are  not 
interested  in  the  cost  or  time/schedule  of  delivering  the  electricity  or  water;  they  want  an 
immediate  and  quality  output.  Hence,  the  Operate  systems  function  intersects  with  the 
quality  and  work  product  delivered  gauge  clusters.  The  remaining  matrix  was  completed 
in  the  same  manner.  The  following  sections  (one  for  each  of  the  seven  functions  found  in 
Table  50)  recommend  a  metric  to  use  in  future  competitive  sourcing  efforts.  The 
recommendations  for  improvement  were  provided  by  1)  slightly  changing  the  content  of 
a  failing  standard  and  metric,  2)  making  changes  based  upon  metrics  found  in  the 
AFCESA  templates,  and  3)  making  changes  based  upon  AFCESA  Operations  Flight  metrics. 
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Operate  Systems 


The  specific  function  labeled  Operate  systems  requires  two  metrics  to  evaluate  the 
areas  from  base  personnel  viewpoint,  which  are  identified  by  the  “X”  markings  in  Table 
50.  The  two  metrics  need  to  evaluate  quality  and  work  product  delivered.  The  two  gauge 
clusters  are  listed  below  with  their  proposed  metrics.  The  leadership  from  the  base  must 
decide  the  priority  of  the  systems.  For  example,  water  treatment  and  electrical 
distribution  may  be  ranked  a  higher  priority  than  air  conditioning,  because  water  and 
electricity  may  have  much  more  impact  on  the  mission  than  air  conditioning.  Metrics 
can  be  developed  according  to  the  priority  of  the  systems.  The  metrics  provided  in  this 
research  effort  do  not  account  for  priority  among  the  systems. 

Quality 


The  proposed  quality  metric  was  adapted  from  line  #64  (1.1  Operations  Flight 
Function),  M-38  (Operations  Flight  metrics),  and  T-3  (AFCESA  templates).  Table  51, 
Proposed  Operate  Systems  Quality  Metric,  lists  the  objective,  threshold  and  surveillance 
for  the  proposed  metric,  which  focuses  on  the  quality  of  the  output  from  the  system. 


Table  51.  Proposed  Operate  Systems  Quality  Metric 


Objective 

Threshold 

Surveillance 

Systems  output  commensurate 
with  industry  standards 

95%  of  all  systems  (as 
determined  by  leadership) 
measured  weekly  must  have 
desired  output  (or  a  frequency 
established  bv  leadership) 

Review  of  records  and  customer 
complaints  on  a  weekly  basis 
(must  match  threshold  time) 
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Work  Product  Delivered 


The  proposed  work  product  delivered  metric  was  adapted  from  line  #12  (1.5 
Operations  Flight  Function),  M-41  and  M-42  (Operations  Flight  metrics),  and  T-4 
(AFCESA  templates).  Table  52,  Proposed  Operate  Systems  Work  Product  Delivered 
Metric,  lists  the  objective,  threshold  and  surveillance  for  the  proposed  metric,  which 
focuses  on  the  output  from  the  system. 


Table  52.  Proposed  Operate  Systems  Work  Product  Delivered  Metric 


Objective 

Threshold 

Surveillance 

Operate  system 

System  mission  impacting  failure 
time  will  not  exceed  5.3  minutes 
(99.999%)  annually  (or  other 
established  time) 

Monthly  review  of  records  and 
customer  complaints  (time  must 
match  threshold  time) 

Maintain  Systems 

The  specific  function  labeled  Maintain  Systems  requires  four  metrics  to  evaluate 
the  areas  from  base  personnel  viewpoint,  which  are  identified  by  the  “X”  markings  in 
Table  50.  The  four  metrics  need  to  evaluate  time/schedule,  quality,  finance/budget,  and 
work  product  delivered.  The  four  gauge  clusters  are  listed  below  with  their  proposed 
metrics.  The  maintenance  of  systems  would  also  require  a  method  to  prioritize  the 
systems  according  to  base  mission. 
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Time/Schedule 


The  proposed  time/schedule  metric  was  adapted  from  line  #6  (1.1  Operations  Flight 
Function),  M-1  through  M-5  (Operations  Flight  metrics),  and  T-2  (AFCESA  templates). 
Table  53,  Proposed  Maintain  Systems  Time/Schedule  Metric,  lists  the  objective,  threshold 
and  surveillance  for  the  proposed  metric,  which  focuses  on  the  response  time  for 
maintaining  the  system.  The  time/schedule  metrics  should  contain  three  subsets: 
Emergency,  Elrgent,  and  Routine,  because  each  of  the  subsets  contains  different  time 
response  criteria. 


Table  53.  Proposed  Maintain  Systems  Time/Schedule  Metric 


Objective 

Threshold 

Surveillance 

Emergency  (substitute  urgent  or 
routine)  work  requests  completed 
in  24  hours  (substitute  5  days  and 
30  days  for  urgent  and  routine, 
respectively) 

95%  of  all  systems  (as 
determined  by  leadership) 
measured  weekly  must  have 
desired  maintencance  (or  a 
frequency  established  by 
leadership) 

Review  of  records  and  customer 
complaints  on  a  weekly  basis 
(must  match  threshold  time) 

Quality 


The  proposed  quality  metric  was  adapted  from  line  #30  (1.1  Operatbns  Flight 
Function),  and  T-3  (AFCESA  template).  Table  54,  Proposed  Maintain  Systems  Quality 
Metric,  lists  the  objective,  threshold  and  surveillance  for  the  proposed  metric,  which 
focuses  on  the  quality  of  the  system  maintenance. 


99 


Table  54.  Proposed  Maintain  Systems  Quality  Metric 


Objective 

Threshold 

Surveillance 

Maintain  system  to  standard 
commensurate  with  design 
criteria  and  accepted  industry 
standards 

95%  of  all  systems  (as 
determined  by  leadership) 
measured  weekly  must  have 
desired  maintenance  (or  a 
frequency  established  by 
leadership) 

Random  sampling,  review  of 
records  (parts  consumed)  and 
customer  complaints  on  a  weekly 
basis  (must  match  threshold  time) 

Finance/Budget 


The  proposed  fmance/budget  metric  was  adapted  from  line  #59  (1.7  Operations 
Flight  Function),  and  T-3  (AFCESA  template).  Table  55,  Proposed  Maintain  Systems 
Finance/Budget  Metric,  lists  the  objective,  threshold  and  surveillance  for  the  proposed 
metric,  which  focuses  on  the  cost  of  maintaining  the  system. 


Table  55.  Proposed  Maintain  Systems  Finance/Budget  Metric 


Objective 

Threshold 

Surveillance 

Measure  the  cost  of  maintaining 
system 

Determine  some  annual 
percentage  of  the  total  cost  of  the 
system  that  the  maintenance 
cannot  exceed  (identifies  aging 
systems) 

Annual  review  of  maintenance 
records  and  repair  actions  (time 
must  match  threshold  time) 

Work  Product  Delivered 


The  proposed  work  product  delivered  metric  was  adapted  from  line  #12  (1.5 
Operations  Flight  Function),  M-6  (Operations  Flight  metrics),  and  T-4  (AFCESA 
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template).  Table  56,  Proposed  Maintain  Systems  Work  Product  Delivered  Metric,  lists 
the  objective,  threshold  and  surveillance  for  the  proposed  metric,  which  focuses  on  the 
maintenance  of  the  system. 


Table  56.  Proposed  Maintain  Systems  Work  Product  Delivered  Metric 


Objective 

Threshold 

Surveillance 

Maintain  system  according  to 
schedule 

95%  of  all  systems  (as 
determined  by  leadership) 
measured  weekly  must  have 
desired  maintenance  (or  a 
frequency  established  by 
leadership) 

Monthly  review  of  maintenance 
records  and  repair  actions  (time 
must  match  threshold  time) 

Repair  Systems 

The  specific  function  labeled  Repair  systems  requires  four  metrics  to  evaluate  the 
areas  from  base  personnel  viewpoint,  which  are  identified  by  the  “X”  markings  in  Table 
50.  The  four  metrics  need  to  evaluate  time/schedule,  quality,  fmance/budget,  and  work 
product  delivered.  The  four  gauge  clusters  are  listed  below  with  their  proposed  metrics. 

It  was  found  that  the  maintenance  metrics  could  serve  as  repair  metrics  as  well,  just  by 
replacing  maintain  with  repair  in  the  metrics. 

Time/Schedule 


The  proposed  time/schedule  metric  was  adapted  from  line  #6  (1.1  Operations  Flight 
Function),  M-1  through  M-5  (Operations  Flight  metrics),  T-2  (AFCESA  template).  Table 
57,  Proposed  Maintain  Systems  Time/Schedule  Metric,  lists  the  objective,  threshold  and 
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surveillance  for  the  proposed  metric,  which  focuses  on  the  response  time  for  repairing  the 


system. 


Table  57.  Proposed  Repair  Systems  Time/Schedule  Metric 


Objective 

Threshold 

Surveillance 

Emergency  (substitute  urgent  or 
routine)  work  requests  completed 
in  24  hours  (substitute  5  days  and 
30  days  for  urgent  and  routine, 
respectively) 

95%  of  all  systems  (as 
determined  by  leadership) 
measured  weekly  must  have 
desired  response  timet  (or  a 
frequency  established  by 
leadershio) 

Review  of  records  and  customer 
complaints  on  a  weekly  basis 
(must  match  threshold  time) 

Quality 


The  proposed  quality  metric  was  adapted  from  line  #30  (1.1  Operations  Flight 
Function)  and  T-3  (AFCESA  template).  Table  58,  Proposed  Repair  Quality  Metric,  lists 
the  objective,  threshold  and  surveillance  for  the  proposed  metric,  which  focuses  on  the 
quality  of  the  system  repair. 


Table  58.  Proposed  Repair  Quality  Metric 


Objective 

Threshold 

Surveillance 

Repair  system  to  standard 
commensurate  with  design 
criteria  and  accepted  industry 
standards 

95%  of  all  systems  (as 
determined  by  leadership) 
measured  weekly  must  be 
repaired  according  to  standards 
(or  a  frequency  established  by 
leadership) 

Random  sampling,  review  of 
records  (parts  consumed)  and 
customer  complaints  on  a  weekly 
basis  (must  match  threshold  time) 
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Finance/Budget 


The  proposed  f^nance^udget  metric  was  adapted  from  line  #59  (1.7  Operations 
Flight  Function)  and  T-4  (AFCESA  template).  Table  59,  Proposed  Repair  Systems 
Finance/Budget  Metric,  lists  the  objective,  threshold  and  surveillance  for  the  proposed 
metric,  which  focuses  on  the  cost  of  repairing  the  system. 


Table  59.  Proposed  Repair  Systems  Finance/Budget  Metric 


Objective 

Threshold 

Surveillance 

Measure  the  cost  of  repairing 
system 

Determine  some  annual 
percentage  of  the  total  cost  of  the 
system  that  the  repairs  cannot 
exceed  (identifies  aging  systems) 

Annual  review  of  repair  records 
and  repair  actions  (time  must 
match  threshold  time) 

Work  Product  Delivered 


The  proposed  work  product  delivered  metric  was  adapted  from  line  #12  (1.5 
Operations  Flight  Function),  M-13  (Operations  Flight  metrics)  and  T-3  (AFCESA 
template).  Table  60,  Proposed  Repair  Systems  Work  Product  Delivered  Metric,  lists  the 
objective,  threshold  and  surveillance  for  the  proposed  metric,  which  focuses  on  the 
completion  of  the  repair  on  the  system. 
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Table  60.  Proposed  Repair  Systems  Work  Product  Delivered  Metric 


Objective 

Threshold 

Surveillance 

Repair  system  according  to 
schedule 

95%  of  all  systems  (as 
determined  by  leadership) 
measured  weekly  must  be 
repaired  according  to  standards 
(or  a  frequency  established  by 
leadership) 

Monthly  review  of  maintenance 
records  and  repair  actions  (time 
must  match  threshold  time) 

Installation  Support 

The  specific  function  labeled  installation  support  requires  three  metrics  to 
evaluate  the  areas  from  base  personnel  viewpoint,  which  are  identified  by  the  “X” 
markings  in  Table  50.  The  three  metrics  need  to  evaluate  time/schedule,  quality,  and 
work  product  delivered.  The  three  gauge  clusters  are  listed  below  with  their  proposed 
metrics.  The  finance/budget  gauge  cluster  is  not  used  because  the  base  personnel  may 
not  be  interested  in  the  cost  of  providing  the  support,  they  are  more  interested  in  results. 

Time/Schedule 


The  proposed  time/schedule  metric  was  adapted  from  M-13  (Operations  Flight 
Metrics)  and  T-6  (AFCESA  template).  Table  61,  Proposed  Installation  Support 
Time/Schedule  Metric,  lists  the  objective,  threshold  and  surveillance  for  the  proposed 
metric,  which  focuses  on  the  response  time  for  installation  support  (e.g.,  pest 
management,  and  snow  removal). 
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Table  61.  Proposed  Installation  Support  Time/Schedule  Metric 


Objective 

Threshold 

Surveillance 

Measure  the  number  of  base 
support  commitments 

(number  of  commitments 
completed  on  time  divided  by  the 
number  of  commitments) 

95%  of  all  support  commitments 
(as  determined  by  leadership) 
measured  monthly  must  be 
completed  on  time  (or  a 
frequency  established  by 
leadership) 

Review  of  records  and  customer 
complaints  on  a  monthly  basis 
(must  match  threshold  time) 

Quality 


The  proposed  quality  metric  was  adapted  from  line  #30  (1.1  Operations  Flight 
Function.  Table  62,  Proposed  Installation  Support  Quality  Metric,  lists  the  objective, 
threshold  and  surveillance  for  the  proposed  metric,  which  focuses  on  the  quality  of  the 
support  received. 


Table  62.  Proposed  Installation  Quality  Metric 


Objective 

Threshold 

Surveillance 

Installation  support 
commensurate  with  industry 
practices 

95%  of  all  support  (as  determined 
by  leadership)  measured  monthly 
must  be  completed  according  to 
standards  (or  a  frequency 
established  bv  leadership) 

Random  sampling,  review  of 
records  and  customer  complaints 
on  a  monthly  basis  (must  match 
threshold  time) 

Work  Product  Delivered 


The  proposed  work  product  delivered  metric  was  adapted  from,  M- 1  through  M- 
5,  and  M-13  (Operations  Flight  Metrics),  and  T-6  (AFCESA  template).  Table  63, 
Proposed  Installation  Support  Work  Product  Delivered  Metric,  lists  the  objective. 
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threshold  and  surveillance  for  the  proposed  metric,  which  focuses  on  the  output  from  the 
installation  support. 


Table  63.  Proposed  Installation  Work  Product 

Delivered  Metric 

Objective 

Threshold 

Surveillance 

Measure  the  number  of 
commitments 

(number  of  commitments 
completed  divided  by  number  of 
total  commitments) 

95%  of  all  support  commitments 
(as  determined  by  leadership) 
measured  monthly  must  be 
completed  (or  a  frequency 
established  by  leadership) 

Monthly  review  of  records  (time 
must  match  threshold  time) 

Self-Help 

The  specific  function  labeled  self-help  requires  three  metrics  to  evaluate  the  areas 
from  base  personnel  viewpoint,  which  are  identified  by  the  “X”  markings  in  Table  50. 

The  three  metrics  need  to  evaluate  time/schedule,  quality,  and  work  product  delivered. 
The  three  gauge  clusters  are  listed  below  with  their  proposed  metrics.  The 
fmance/budget  gauge  cluster  is  not  used  because  the  base  personnel  may  not  be  interested 
in  the  cost  of  providing  the  self-help,  they  are  more  interested  in  results. 

Time/Schedule 


The  proposed  time/schedule  metric  was  adapted  from  M-13  (Operations  Flight 
Metrics)  and  T-6  and  T-7  (AFCESA  template).  Table  64,  Proposed  Installation  Support 
Time/Schedule  Metric,  lists  the  objective,  threshold  and  surveillance  for  the  proposed 
metric,  which  focuses  on  the  response  time  for  self-help. 
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Table  64.  Proposed  Self-Help  Time/Schedule  Metric 


Objective 

Threshold 

Surveillance 

Measure  the  number  of  self-help 
commitments  completed  on  time 

(number  of  commitments 
completed  on  time  divided  by  the 
number  of  commitments) 

95%  of  self-help  commitments 
(as  determined  by  leadership) 
measured  monthly  must  be 
delivered  on  time  (or  a  frequency 
established  by  leadership) 

Review  of  records  and  customer 
complaints  on  a  monthly  basis 
(must  match  threshold  time) 

Quality 


The  proposed  quality  metric  was  adapted  from  line  #23  (1.12  Operations  Flight 
Function)  and  T-7  (AFCESA  template).  Table  65,  Proposed  Self-Help  Quality  Metric, 
lists  the  objective,  threshold  and  surveillance  for  the  proposed  metric,  which  focuses  on 
the  quality  of  the  self-help  received. 


Table  65.  Proposed  Self-Help  Quality  Metric 


Objective 

Threshold 

Surveillance 

Track  variance  in  cost  estimates. 

Conduct  analysis  of  all  estimates 
having  a  10%  or  greater 
difference  between  planned  and 
actual  cost) 

95%  of  self-help  commitments(as 
determined  by  leadership) 
measured  monthly  must  not  be 
greater  than  1 0%  variance  (or  a 
frequency  established  by 
leadership) 

Random  sampling,  review  of 
records  and  customer  complaints 
on  a  monthly  basis  (must  match 
threshold  time) 
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Work  Product  Delivered 


The  proposed  work  product  delivered  metric  was  adapted  from  M-13  (Operations 
Flight  Metrics)  and  T-7  (AFCESA  template).  Table  66,  Proposed  Self-Help  Work 
Product  Delivered  Metric,  lists  the  objective,  threshold  and  surveillance  for  the  proposed 
metric,  which  focuses  on  the  output  from  the  self-help. 


Table  66.  Proposed  Self-Help  Product  Delivered  Metric 


Objective 

Threshold 

Surveillance 

Measure  the  number  of  self-help 
customers 

(number  of  self-help  estimates 
completed  divided  by  number  of 
total  self-helo  estimates) 

95%  of  all  self-help  commitments 
(as  determined  by  leadership) 
measured  monthly  must  be 
completed  (or  a  frequency 
established  by  leadership) 

Monthly  review  of  records  and 
customer  complaints  (time  must 
match  threshold  time) 

Facility  Mana2er  Plan 

The  specific  function  labeled  facility  manager  plan  requires  three  metrics  to 
evaluate  the  areas  from  base  personnel  viewpoint,  which  are  identified  by  the  “X” 
markings  in  Table  50.  The  three  metrics  need  to  evaluate  time/schedule,  quality,  and 
work  product  delivered.  The  three  gauge  clusters  are  listed  below  with  their  proposed 
metrics.  The  finance/budget  gauge  cluster  is  not  used  because  the  base  personnel  may 
not  be  interested  in  the  cost  of  providing  the  facility  plans,  they  are  more  interested  in 
results. 
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Time/Schedule 


The  proposed  time/schedule  metric  was  adapted  from  M-1  through  M-5  and  M-13 
(Operations  Flight  Metrics)  and  T-7  (AFCESA  template).  Table  67,  Proposed  Facility 
Manager  Plan  Time/Schedule  Metric,  lists  the  objective,  threshold  and  surveillance  for 
the  proposed  metric,  which  focuses  on  the  response  time  for  providing  plans  and  updates. 


Table  67.  Proposed  Facility  Manager  Plan  Time/Schedule  Metric 


Objective 

Threshold 

Surveillance 

Measure  the  number  of  plans  and 
updates 

(number  of  plans  and  updates 
completed  on  time  divided  by  the 
number  of  olans  and  uodates) 

95%  of  all  plans  and  updates  (as 
determined  by  leadership) 
measured  monthly  must  be 
completed  on  time  (or  a 
frequency  established  by 
leadership) 

Review  of  records  and  customer 
complaints  on  a  monthly  basis 
(must  match  threshold  time) 

Quality 


The  proposed  quality  metric  was  adapted  from  line  #23  (1.12  Operations  Flight 
Function)  and  T-7  (AFCESA  template).  Table  68,  Proposed  Facility  Manager  Plans 
Quality  Metric,  lists  the  objective,  threshold  and  surveillance  for  the  proposed  metric, 
which  focuses  on  the  quality  of  the  plans  and  updates. 
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Table  68.  Proposed  Facility  Manager  Plans  Quality  Metric 


Objective 

Threshold 

Surveillance 

Track  variance  in  estimates. 
Conduct  analysis  of  all  estimates 
having  a  10%  or  greater 
difference  between  planned  and 
actual  cost) 

Or  number  of  re  -writes 

95%  of  all  plans  and  updates  (as 
determined  by  leadership) 
measured  monthly  cannot  exceed 
10%  variance  between  planned 
and  actual  cost  (or  a  frequency 
established  by  leadership) 

Random  sampling,  review  of 
records  and  customer  complaints 
on  a  monthly  basis  (must  match 
threshold  time) 

Work  Product  Delivered 


The  proposed  work  product  delivered  metric  was  adapted  from  line  #12  (1.5 
Operations  Flight  Function),  M-1  through  M-5  (Operations  Flight  metrics),  and  T-7 
(AFCESA  template).  Table  69,  Proposed  Facility  Manager  Plan  Work  Product 
Delivered  Metric,  lists  the  objective,  threshold  and  surveillance  for  the  proposed  metric, 
which  focuses  on  the  completion  of  plans  and  updates. 


Table  69.  Proposed  Facility  Manager  Plan  Work  Product  Delivered  Metric 


Objective 

Threshold 

Surveillance 

Measure  the  number  of  plans  and 
updates 

(number  of  plans  and  updates 
completed  divided  by  number  of 
total  plans  and  updates) 

95%  of  all  plans  and  updates  (as 
determined  by  leadership) 
measured  monthly  must  be 
completed  (or  a  frequency 
established  by  leadership) 

Monthly  review  of  records  and 
customer  complaints  (time  must 
match  threshold  time) 

The  19  proposed  metrics  cover  all  aspects  of  the  consolidated  Operations  Flight 
Functions  and  the  gauge  clusters,  given  that  base  leadership  prioritizes  the  systems.  The 
proposed  metrics  were  developed  using  input  from  the  eight  bases,  the  AFCESA 
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templates,  and  the  AFCESA  Operations  Flight  Metrics.  AFCESA  should  incorporate  the 
proposed  metrics  into  the  Service  Delivery  Summary  of  the  PWS  and  QASP  templates. 
Better  produced  performance  metrics  will  result  at  base  level. 

Results 

Based  upon  the  results  of  this  research,  several  findings  evolved.  Five  findings 
will  be  listed  and  discussed  in  the  following  sections.  The  first  finding  is  that  some  bases 
do  not  use  the  QASP  document.  Personnel  from  those  bases  stated  that  a  QASP  was  not 
needed  at  the  time  the  competitive  sourcing  process  took  place,  but  their  PWS  stated  that 
a  QASP  must  be  followed.  Another  base  that  did  not  have  any  lines  pass  the  primary 
evaluation  did  not  use  a  QASP  either.  The  requirement  for  a  QASP,  outlined  in  AFI  63- 
124,  is  not  enforced  at  base  level.  This  results  in  an  improper  evaluation  to  determine  if 
the  service  provider  has  managed  to  maintain  or  increase  efficiency. 

The  second  finding  comes  from  the  lines  that  passed  the  secondary  evaluation. 

The  six  lines  targeted  the  time/schedule  gauge  cluster,  leaving  many  of  the  gauge  clusters 
unevaluated.  There  are  many  of  the  gauge  clusters  identified  in  Chapter  2,  Literature 
Review  that  are  not  properly  evaluated  or  not  evaluated  at  all.  Civil  engineering  is  a 
customer  service  operation  and  should  focus  its  efforts  on  customer  satisfaction  (although 
very  difficult  to  quantify),  finance^udget  (doing  the  work  in  the  most  cost  effective 
manner),  and  quality  (doing  the  work  right  the  first  time).  These  three  gauge  clusters 
(customer  satisfaction,  finance^udget,  and  quality)  focus  the  standards  and  metrics  to 
evaluate  aspects  that  are  important  to  the  customer  (base  personnel).  The  lines  lacked 
many  of  the  other  gauge  clusters  as  well.  For  instance,  work  product  delivered,  business 
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value,  operational  service  levels,  human  resources,  and  productivity  gauge  clusters  were 
not  properly  evaluated  or  were  not  included  at  all  in  the  standards  and  metrics. 

The  third  finding  is  that  the  critical  areas  defined  by  the  Operations  Flight 
Functions  are  not  sufficiently  evaluated.  Many  of  the  standard  and  metric  lines  taken 
from  the  competitive  sourcing  documents  were  too  broad  and  not  understandable  or 
quantifiable,  leaving  critical  areas  without  evaluation.  There  were  only  three  Operations 
Flight  Functions  that  have  some  properly  designed  metrics,  based  upon  results  of  this 
research.  Six  metrics  were  found  to  sufficiently  evaluate  Operations  Flight  Function  1.6, 
Base  Support,  Function  1.3,  Emergency  Response,  and  1.1,  Operate,  Maintain,  and 
Repair.  Not  all  gauge  clusters  were  found  in  the  six  lines  that  passed  both  the  primary 
and  secondary  evaluation;  five  of  the  Operations  Flight  Functions  had  the  time/schedule 
gauge  cluster  and  one  had  the  Operational  Service  Level  gauge  cluster.  When  the  seven 
gauge  clusters  are  applied  to  the  14  Operations  Flight  Functions,  it  is  apparent  that  none 
of  the  14  functions  are  properly  evaluated. 

The  fourth  finding  involves  the  competitive  sourcing  classification  and 
competitive  sourcing  databases.  Numerous  errors  of  improperly  labeled  contracts  were 
identified  in  AFCESA  databases  at  the  outset  of  this  research.  For  example,  a  base  was 
identified  as  having  two  contracts,  one  an  MEO  and  the  other  a  private  sector.  After 
contacting  the  base,  it  was  discovered  that  the  private  sector  contract  was  non-existent. 
The  other  example  was  mentioned  in  Chapter  3,  Methodology,  in  which  the  base  was 
identified  as  having  a  BOS  contract,  but  after  contacting  the  base,  it  was  discovered  the 
base  had  two  small  MEO  contracts  and  two  small  private  sector  contracts. 
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The  fifth  finding  of  this  research  is  the  lack  of  properly  designed  standards  and 
metric  provided  as  examples  in  the  templates  and  list  of  metrics.  Because  of  the 
extensive  efforts  in  competitive  sourcing,  bases  would  benefit  from  quality  examples  to 
guide  them  in  the  proper  measurement  and  evaluation  of  the  effort  that  is  being 
competitively  sourced.  The  examples  of  standards  and  metrics  provided  are  poorly 
designed  to  properly  evaluate  the  competitive  sourcing  effort  from  the  base  personnel 
viewpoint.  They  do  provide  some  insight  as  to  what  to  look  for  when  designing  the 
metric,  but  they  should  not  be  used  as  they  are  written.  If  quality  standards  and  metrics 
are  not  used,  sustained  or  improved  efficiency  of  competitive  sourcing  efforts  cannot  be 
determined,  resulting  in  wasted  money  and  personnel  time  to  competitively  source  a 
function  that  may  or  may  not  be  improved,  regardless  of  who  provides  the  service.  The 
lines  used  in  the  competitive  sourcing  efforts  of  this  research  are  not  sufficient  to  provide 
information  if  the  service  provider  is  maintaining,  improving,  or  declining  in 
performance.  Although,  some  of  the  lines  not  containing  a  frequency  is  an  easy  fix,  if  the 
only  step  the  line  failed  was  the  lack  of  frequency  and  the  frequency  is  the  correct 
frequency  for  the  process.  Overall,  because  of  the  lack  of  performance  data,  it  is  not 
proven  that  competitive  sourcing  is  a  method  to  improve  or  maintain  efficiency. 

Summary 

In  summary,  of  the  approximately  1500  commercial  activities  in  the  Air  Force, 
nearly  one- third  is  civil  engineering  related.  This  research  selected  10  (later  reduced  to 
eight)  bases  to  provide  civil  engineering  competitive  sourcing  documents.  From  the  eight 
bases,  a  variety  of  standards  and  metrics  were  collected,  161  total,  and  the  variety 
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indicates  a  non- standardized  method  of  creating  standards  and  metrics,  even  with  the 
templates  and  metrics  as  guidelines.  Each  of  the  161  lines  were  then  subjected  to  a 
primary  line  evaluation  to  determine  if  the  standard  and  metric  were  properly  designed. 
Only  seven  standards  and  metrics  passed  the  primary  line  evaluation.  A  secondary  line 
evaluation  (quantitative)  was  then  applied  to  the  seven  standard  and  metric  lines  that 
passed  the  primary  line  evaluation.  The  secondary  line  evaluation  consisted  of 
telecommunications  with  respective  base  personnel.  Six  standard  and  metric  lines  passed 
the  secondary  line  evaluation;  only  4%  of  the  standards  and  metrics  evaluated  in  this 
research  are  properly  designed. 

This  chapter  discussed  the  primary  and  secondary  evaluations  of  the  performance 
standard  and  metric  lines.  Each  of  the  161  lines  was  evaluated  and  example  lines  were 
provided  for  each  of  the  steps  that  failed  during  the  evaluations.  This  chapter  also 
provided  an  overview  of  the  AFCESA  PWS  template  and  the  Operations  Flight  Metrics. 
Metrics  were  proposed  for  use  in  future  competitive  sourcing  efforts  and  were  developed 
by  adapting  the  metrics  found  on  the  templates,  list  of  AFCESA  metrics,  and  the 
performance  standards  and  metrics  found  within  conpetitive  sourcing  documents. 

Finally,  the  chapter  closed  with  a  brief  discussion  of  the  results. 
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5.  Conclusions  and  Recommendations 


This  chapter  will  provide  a  summary  of  the  research  effort  and  the  major  findings. 
Limitations  of  this  research  are  discussed,  as  are  future  research  topics.  Finally,  this 
chapter  will  provide  recommendations  based  upon  the  results  of  the  data  collection  and 
the  evaluation  of  the  performance  standards  and  metrics. 

Summary  of  Research  Effort 

The  purpose  of  this  research  effort  was  to  evaluate  standards  and  metrics  currently 
used  in  civil  engineering  competitive  souring  efforts.  An  evaluation  system  was 
established  and  then  applied  to  the  standards  and  metrics,  resulting  in  the  identification  of 
improperly  designed  metrics  and  critical  areas  not  evaluated.  This  research  consisted  of 
obtaining  competitive  sourcing  documents  from  eight  bases  from  various  Air  Force 
Commands.  An  attempt  was  made  to  collect  documents  from  ten  bases;  two  of  the  bases 
considered  had  either  an  improperly  labeled  contract  or  had  pulled  their  solicitation. 

Once  the  documents  were  collected,  the  standards  and  metrics  were  extracted  from  the 
Service  Delivery  Summary,  or  equivalent,  (the  portion  of  the  PWS  that  lists  the  services 
of  the  contract),  and  merged  into  lines  that  contained  the  objective,  threshold,  and  the 
surveillance.  A  total  of  161  lines  were  created  from  the  documents  collected  from  the 
eight  bases.  Each  line  was  then  identified  for  its  respective  Operations  Flight  Function 
and  its  corresponding  gauge  cluster.  Once  the  lines  were  identified,  they  were  evaluated 
based  upon  a  system  created  by  features  from  metric  design  literature.  Total  Quality 
Management,  and  Government  Performance  Results  Act.  A  primary  evaluation  was 
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established  from  six  of  the  features  and  a  secondary  evaluation  was  established  from  five 
of  the  features.  The  researcher  reviewed  each  of  the  lines  to  conduct  the  primary  line 
evaluation.  The  secondary  line  evaluation  involved  features  that  only  the  specific  bases 
could  answer  and  the  information  obtained  from  the  bases  provided  a  quantitative 
secondary  line  evaluation.  Only  33  lines  passed  the  primary  evaluation  to  continue  with 
the  secondary  evaluation.  Some  of  the  standards  and  metrics  are  easy  to  fix  (i.e.,  add 
frequency  or  sensor)  but  others  require  major  work  because  they  are  too  broad  (not 
specific  in  what  information  is  important),  or  they  have  a  mismatch  between  the 
objective,  threshold,  or  surveillance.  26  of  the  33  metrics  were  not  evaluated  because 
some  bases  did  not  have  QASPs  and  others  had  not  initiated  a  contract  at  the  time  of  the 
research,  therefore,  only  seven  lines  were  evaluated  by  the  secondary  evaluation.  Of  the 
seven  lines  evaluated  in  the  secondary  evaluation,  six  were  determined  as  sufficiently 
designed  (passed  both  the  primary  and  secondary  evaluations  with  a  “yes”  to  all  eleven 
questions). 

Summary  of  Conclusions 

Once  the  lines  were  created  and  just  prior  to  the  primary  evaluation,  it  was 
evident  that  critical  areas  of  the  Operations  Flight  were  not  being  sufficiently  evaluated. 
In  fact,  some  areas  were  not  evaluated  at  all,  for  example,  “develop  and  update  future 
plans”,  “provide  customers  with  costs  of  work  or  services”,  and  “provide  and  effective 
facility  manager  program”  were  three  Operations  Flight  Functions  that  were  not 
evaluated.  The  remaining  1 1  Operations  Flight  Functions  were  insufficiently  evaluated 
because  128  lines  failed  some  aspect  of  the  primary  evaluation. 
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Each  line  was  classified  according  to  a  gauge  cluster.  This  provided  information 
that  the  competitive  sourcing  documents  utilized  few  of  the  nine  gauge  clusters.  Some  of 
the  nine  gauge  clusters  were  absent  from  the  beginning  of  the  research.  Very  few  of  the 
lines  contained  a  gauge  classification  of  quality,  finance/budget,  productivity,  work 
product  delivered,  customer  satisfaction,  and  business  value,  and  none  of  the  lines 
contained  a  gauge  classification  of  human  resources.  Quality  and  Customer  Service 
should  be  integral  to  the  evaluation  of  the  civil  engineering  processes  because  civil 
engineering  is  a  service,  but  quality  and  customer  service  are  very  difficult  to  quantify. 

Other  findings  from  the  research  are  that  templates  and  current  Operations  Flight 
Metrics,  for  the  most  part,  are  not  sufficient  to  evaluate  civil  engineering  (based  upon 
base  personnel  as  the  customer).  The  metrics  need  to  be  reassessed  to  ensure  civil 
engineering  units  are  improving  in  their  provided  services  of  quality,  cost,  and  customer 
satisfaction.  Due  to  the  number  of  bases  selected,  it  was  also  found  that  many  of  the 
competitive  sourcing  documents  lacked  similarity.  The  bases  from  within  the  same 
command  typically  had  similar  formats,  but  the  formats  varied  widely  from  command  to 
command.  Even  the  bases  that  are  now  undergo  ing  the  competitive  sourcing  process 
have  their  own  format  and  style  for  their  documents.  The  variety  of  documents  created 
some  difficulties  when  initiating  this  research.  Finally,  it  was  found  that  record  keeping 
for  the  competitive  sourcing  efforts  was  not  error-  free;  several  errors  were  found  in  the 
contract  classifications  at  the  outset  of  this  research. 

Bases  need  to  have  enforcement  of  the  QASP  document.  The  QASP  will  ensure 
the  performance  requirements  are  tracked.  Without  the  QASP,  service  providers  are  not 
evaluated  properly  and  sustained  or  improved  performance  cannot  be  determined.  Also, 
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there  are  many  critical  areas  without  sufficient  evaluation.  The  critical  areas  can  be 
readily  identified  (i.e.,  flowcharts)  and  metrics  can  be  developed  with  the  19  proposed 
metrics.  The  19  proposed  metrics  provide  a  solid  foundation  to  build  the  metrics.  Also, 
the  metric  evaluation  system  used  in  this  research  can  be  applied  to  any  metric  in  any 
specialty.  A  “no”  answer  to  any  of  the  questions  indicates  a  poorly  designed  metric. 

Once  the  metric  is  identified  as  insufficient,  closer  inspection  will  identify  what  is  needed 
to  improve  the  metric. 

Research  Limitations 

There  were  a  few  limitations  apparent  from  the  outset  of  this  research.  First,  this 
research  was  limited  because  only  the  Air  Force  was  evaluated.  There  are  a  variety  of 
competitive  sourcing  documents  from  several  Federal  Government  departments  that 
would  provide  insight  into  which  performance  measurement  programs  are  effective. 
Within  the  Air  Force,  documents  were  used  from  only  eight  bases,  which  narrowed  the 
scope  of  the  research.  Second,  the  research  was  limited  because  only  the  Operations 
Flight  of  the  civil  engineering  unit  was  evaluated;  the  scope  of  performance  measurement 
was  limited  to  just  a  portion  of  the  civil  engineering  unit.  A  complete  evaluation  of  the 
entire  civil  engineering  unit  should  be  accomplished  to  effectively  evaluate  the  unit  as  a 
whole,  and  to  develop  quality  metrics.  Finally,  the  research  was  partially  subjective  and 
depended  upon  the  knowledge  of  the  researcher. 
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Future  Research 


If  competitive  sourcing  efforts  continue  as  planned,  there  will  be  an  increased 
need  in  determining  if  the  functions  that  are  being  outsourced  are  maintaining, 
improving,  or  declining  in  performance.  The  only  way  to  determine  if  competitive 
sourcing  is  saving  money  while  providing  a  reasonable  level  of  service  is  to  have 
properly  designed  metrics  evaluating  the  process.  The  focus  of  the  designed  metrics 
should  consider  both  financial  and  non- financial  aspects  of  the  process.  With  this  in 
mind,  there  are  many  other  functions  that  should  have  their  standards  and  metrics 
evaluated.  Other  possible  areas  to  evaluate  are: 

•  Evaluation  of  the  performance  of  existing  metrics  at  bases  (i.e.,  MAJCOM 
metrics  or  competitive  sourcing  metrics) 

•  Evaluation  of  other  Air  Force  civil  engineering  functions  (i.e.,  MAJCOM 
metrics  or  competitive  sourcing  metrics) 

•  Evaluation  of  other  Air  Force  competitive  sourcing  efforts  (communications, 
services,  aircraft  maintenance,  to  name  a  few) 

•  Evaluation  of  other  governmental  agencies  (Army,  Navy,  or  Department  of 
Energy)  metrics  and  their  competitive  sourcing  efforts 

•  Evaluation  of  private  sector  firms  that  have  outsourced  one  or  more  functions 
and  compare  their  standards  and  metrics  to  a  similar  Air  Force  function 

•  Continue  the  quantitative  study  of  the  metric  design  process  presented  in  this 
research  to  determine  the  importance  of  each  of  the  1 1  steps  used  in  the  primary 
and  secondary  evaluations.  The  quantitative  test  developed  could  be  applied  to 
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any  metric  or  standard.  Submit  surveys  to  Quality  Assurance  Evaluators  and 
service  providers. 

Recommendations 

The  first  recommendation  is  that  AFCESA  incorporate  the  proposed  metrics  into 
the  service  dehvery  summary  of  the  templates  for  use  in  future  civil  engineering 
Operations  Flight  competitive  sourcing  efforts.  The  proposed  metrics  offer  a  good 
method  to  identify  critical  areas  that  need  evaluation;  they  standardize  the  metrics  across 
the  Air  Force  (current  variations  in  metrics  would  be  minimized)  and  all  service 
providers  are  evaluated  equally  (MEO  vs.  contractor),  the  quality  of  the  proposed  metrics 
is  higher,  all  critical  aspects  of  the  Operations  Flight  would  be  evaluated,  the 
performance  of  the  service  provider  is  effectively  measured  (personnel  know  what  to 
measure  and  when  to  measure),  and  the  objective,  threshold,  and  surveillance  are 
reasonable  because  they  were  adapted  from  current  standards  and  metrics. 

Instituting  the  proposed  metrics  would  impact  the  Plan  of  Action  and  Milestone 
(POAM)  for  the  process.  Table  1.  The  specific  workload,  data  needed  for  collection,  and 
unit  goals  required  are  identified  in  the  proposed  metric,  resulting  in  less  wasted  effort  in 
collecting  the  necessary  information  to  support  the  metric.  The  proposed  metrics  will 
also  impact  Step  1,  Team  Formation,  of  the  PWS  development  (Figure  2)  because  the 
requirements  of  the  team,  which  will  be  identified  based  upon  the  proposed  metrics, 
should  be  based  upon  the  critical  areas  that  the  proposed  metrics  evaluate.  Also,  the 
databases  used  to  store  the  information  regarding  the  competitive  sourcing  efforts  need  to 
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be  reviewed  for  accuracy.  The  fiill  impact  of  competitive  sourcing  cannot  be  evaluated  if 
erroneous  information  is  contained  in  the  databases. 

All  remaining  competitive  sourcing  metrics  should  be  reviewed  and,  if  necessary, 
redesigned  using  the  methodology  created  by  this  research.  The  gauge  clusters  are 
applicable  because  they  were  developed  for  outsourcing  (private  sector)  and  the  functions 
of  the  unit  can  be  determined  by  Air  Force  Instructions. 

The  method  used  to  evaluate  the  metrics  is  a  effective  process  to  determine  if 
metrics  currently  in  place  are  properly  designed.  The  method  used  in  this  research  effort 
is  not  specific  to  competitive  sourcing  documents  and  can  be  used  on  any  metric  used  by 
the  Air  Force  (e.g.,  command  metrics).  First,  by  determining  which  customer  to  focus  on 
(base  personnel,  MAJCOM,  or  CE  unit),  the  metrics  can  be  reviewed  for  proper  design. 
These  properly  designed  metrics  will  lead  to  a  performance  measurement  program  that 
can  be  used  to  determine  if  the  process  is  sustaining  or  improving  efficiency. 


121 


Appendix  A.  Plan  of  Action  and  Milestone 


Appendix  A  contains  the  entire  table  of  the  Plan  of  Action  and  Milestone  (POAM)  for 
PWS  completionas  it  is  found  in  the  Cost  Comparison  Handbook  #4.  The  POAM  is 
critical  in  determining  the  time  to  gather  workload  data  and  metric  information. 
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Action 


Action 

Day 

Planning 

These  actions  are  initiated 
by  the  CC  Team  leader  and 
then  the  PWS  Team  leader 
assumes  responsibility 

Team  Formation 

Choose  PWS  Team  Leader 

Choose  PWS  Team  Members 

Identification  of  Ad  Hoc  Advisors 

Sign  Appropriate  Forms  (non-disclosure/non-compete) 

1 

Conduct  Initial  Team  Meeting 

1 

Review  of  All  Documents  to  Date 

1-5 

Assignment  of  Facilities,  Equipment  and  Support  Services 

1-5 

Overall  A -76  Training 

8-11 

8-11 

POAM  Development 

12-19 

Review  All  Appropriate  Documents  for  Schedule  Deadlines  and  Related 
Information 

12-14 

Prepare  Draft  POAM  for  Installation  Approval 

15-17 

Revise  and  Submit  POAM  for  Command  Approval,  as  Required 

18-19 

Update  POAM  as  Required 

Monthly 

Develop  PWS  Decision  Plan 

12-19 

Develop  Initial  Plan 

12-18 

Request  Initial  Decisions 

19 

Document  Decisions  as  They  Are  Made 

12  -  CC  decision 

Document  Decisions  as  They  Are  Changed 

1 3  -  CC  decision 

Identify  Decisions  that  Have  Not  Been  Made  that  Are  Delaying  the 
Process 

19  -  CC  decision 

Planning  Data  Collection  and  Analysis 

12-19 

Analysis  of  Required  Data 

12-14 

Identification  of  Required  Data 

12-18 

Identification  of  Where  a  POC  May  Be  Required 

12-18 

Decide  on  Detailed  Data  Collection  Activities  and  Schedules  - 
Identifying  Where  Analysis  and  Detailed  Processes  Are  Required 

17-19 

Identifying  Activity  Goals 

22-42 

Just-In  Time  Training  on  this  Action 

22-23 

Initial  Data  Collection  and  Analysis 

22-29 

Data  to  Identify  Activity  Goals 

22-29 

Other  General  Data  to  Write  the  PWS 

22-29 

Initial  Data  Analysis 

22-31 

Identification  of  the  Purpose 

29-36 

Identification  of  the  Goals 

29-42 

Developing  Performance  Outcomes 

29-56 

Just-In  Time  Training  on  this  Action 

29-30 

Identify  Additional  Data  Collection  and  Analysis 

29-36 

Performance  Outcome  Analysis 

29-56 

Identification  of  Outcome  Owner 

36-56 

Verification  of  Purposes,  Goals  and  Services 

45-56 

Developing  Performance  Measures 

36-73 

Just-In  Time  Training  on  this  Action 

36-37 

Identify  Additional  Data  Collection  and  Analysis 

36-43 

Select  Performance  Measures 

36-73 

Document  Supporting  Data 

43-73 

Identify  Performance  Measure  Owner 

50-73 

Document  Sources  and  Location  of  Data 

50-73 

Document  Methodology  for  Completion  and  Management  of  Data 

50  -  73 

Identify  Source  of  Performance  Measure 

50-73 

Deyelop  Performance  Standards 

36-73 

Quality  Standard 

36-73 

Timeliness  Standard 

36-73 

Quantity  Standard  (Workload) 

36-73 

Establish  Baseline 

50-73 

Verify  Purposes,  Goals  and  Outcomes 

50-73 

Write  the  PWS 

32-150 

Just-In  Time  Training  on  this  Action 

22-23 

Document  the  Requirements 

36-150 

Document  the  Workload  (Quantity  Standards) 

36-150 

Document  the  Standards  (Quality  and  Timeliness) 

36-150 

Document  Other  Conditions  of  Performance 

32-149 

Goyemment  Furnished  Seryices 

32-149 

Goyemment  Furnished  Property 

32-149 

General  Information 

72-  149 

Publications  and  Forms 

88-  149 

Other  Exhibits 

88-  149 

Definitions 

133-  149 

PWS  Team  Proyides  Input  to  Sections  B,  L  and  M 

120  -  release  of  RFP 

Reyiew  and  Reyise  the  PWS,  as  Required 

1 50  -  implementation 

Develop  the  OASP 

120-150 

Just-In  Time  Training  on  this  Action 

121-122 

Identify  Key  Performance  Indicators 

120-  125 

Identify  Information  Sources 

120-  130 

Deyelop  Tools 

120-  135 

Write  the  QASP 

120-  150 

Reyise  the  QASP,  as  Required 

1 5 1  -  implementation 

Prepare  for  the  Next  Competition 

After  the  decision 

(A-76  Cost  Comparison  t 

andbook  #4,  2000,  29-31) 
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Appendix  B.  AFCESA  Operations  Flight  Metrics 


Appendix  B  contains  a  listing  of  the  47  Operations  Flight  metrics  as  identified  by 
AFCESA.  This  listing  is  a  subset  of  all  the  civil  engineering  metrics  (listed  by  flight) 
found  on  the  AFCESA  website. 
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ID 

(M- 

metric) 

METRIC 

How  Measured? 

Lower  Limit 
(LL), 

Baseline 
(BL),  & 

Upper  Limit 
(UL) 

M-1 

Work  Complete 
Emergency 

DSW 

Urgent  DSW 
Routine  DSW 
Measured  WO 

number  of  commitments,  number  completed,  number  of 
DSW  backlogged 

Look  for 

trends 

M-2 

Open  by  category 
Emergency 

Urgent 

Routine 

Measured  WO 

number  opened  by  category 

Seek 

explanation  of 
increasing 
number  of 
open  WO. 

M-3 

Completed  by 
category 
Emergency 

Urgent 

Routine 

Measured  WO 

number  completed  by  category  by  month 

Seek 

explanation 
for  decreasing 
number  of 
completed 

WOs. 

M-4 

DSW 

total  time  to  accomplish  work  x  100% 

LL:  90% 

Responsiveness 

Emergency 

Urgent 

Routine 

total  allowed  by  category 

BL:  100% 

UL:  110% 

E:  24  hrs 

U : 5  days 

R:  30  days 

M-5 

Work  Satisfaction 

number  of  commitments  comnleted  on  time  x  100% 

LL:  60% 

Emergency 

DSW 

Urgent  DSW 
Routine  DSW 

number  of  total  commitments 

BL:  80% 

UL:  100% 

M-6 

Scheduled 
Measured  WOs 

total  number  of  davs  to  comnlete  Wos  x  100% 
total  number  of  days  scheduled  to  complete  WOs 
(estimated  completion  date  -  start  date) 

LL:  90% 

BL:  100% 

UL:  110% 

M-7 

WO  life  cycle 
(receipt  to 
completion  by 
category) 
Emergency 

DSW 

Urgent  DSW 
Routine  DSW 

number  of  WOs  in  different  time  frequencies  (0-30 
days,  30-60  days,  etc.)  for  routine  and  in-service  DSW 
&  WO 

Look  at 
increasing 
frequency  of 
WOs  within 
the  different 
time  periods. 

E:  24  hrs 

U:  5  days 

R:  30  days 
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M-8 

Backlog  Total  and 
by  Zone  by 
category 

number  of  WOs  received  vs. 
number  of  WOs  completed 

Look  for 
explanation  of 
increasing 
backlog. 

M-9 

Satisfaction 

Questionnaire  asking  customers  about  service.  Use  7- 
point  Likert  scales  where  1  is  bad  service  and  7  is  good 
service.  Use  approximately  20-25  questions  and  have  at 
least  1/3  of  customers  complete  questionnaire. 

LL:  average 
of  questions  = 

1 

BL:  average 
of  questions  = 

4 

UL:  average 
of  questions  = 

7 

M-10 

RWP  Hrs  by 

number  of  hours  expended  x  100% 

LL:  90% 

Zone/Horizontal 

number  of  hours  scheduled 

BL:  100% 

UL:  110% 

M-11 

RWP  Schedule 

number  of  RWP  <;pmpl?t?(j  x  100% 

LL:  90% 

Effectiveness  by 
Zone/Horizontal 

number  of  RWP  scheduled 

BL:  100% 

UL:  110% 

M-12 

RWP  Items 
completed  by 
month 

number  of  RWP  items  completed 

Look  for 
decreasing 
number  of 

RWP  items 

M-13 

Pest  Management 

number  of  commitments  comnleted  on  time 

LL:  90% 

number  of  commitments 

BL:  100% 

UL:  110% 

M-14 

All  type  WO  -  job 
stoppage 
customer  delays 
awaiting 
materials 
awaiting  man¬ 
hours 

WO  time  line  plotted  and  gaps  identified  for  job 
stopages  with  reason  (need  to  get  an  average  over  all 
WOs) 

Look  at  long 
work 

stoppage  and 
explanations 
for  stops 

M-15 

Planning  metric 

number  of  estimated  hours  x  100% 

LL:  90% 

(Workhour 
Variance:  Est  vs. 
Actual  Work 

Hours) 

actual  work  hours 

BL:  100 

UL:  110% 

M-16 

Delinquent  WO 

number  of  WOs  not  comoleted  allowed  time  x  100% 

LL:  0% 

by  work  order 
category 

Emergency 

DSW 

number  of  WO  in  category  for  the  month 

BL:  10% 

UL:  25% 
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Urgent  DSW 
Routine  DSW 


Material  Received 
on  Time  by 
Category 
Emergency 
Urgent 
Routine 


Number  of 
Material 
Complete  WO 


Number  of  Days 

Material 

Complete 


Bench  Stock 
Availability 


Material  Control 
Metric 

Time  from  1st 
ordered  to  last 
received 


Material  Lead 
Time 


Residue  Material 


number  of  WOs  matenal  complete  within  allowed  timexl00%  lL:  0% 
number  of  WOs  in  category  for  the  month  BL:  10% 

UL:  25% 


number  of  WOs  that  are  material  complete  based  on 
material  completion  date,  number  of  estimated  hours  for 
WO 


number  of  WOs  broken  into  number  of  days  categories 
(0-30  days,  30-60  days,  etc.) 


number  of  days  available  in  bin  for  month  x  100% 


total  number  of  bin  days 


[(item  with  earliest  bill  of  material  order  date)  -  (item 
with  latest  received  date)]  averaged  across  all  WO 


count  number  of  WOs  using  date  of  last  item  received 
stratified  by  age:  0-30  days,  31-60  days,  etc. 


$  value  of  material  in  residue  plotted  against  previous 
FY  average 


Planning  Metric 
(Amt  of  material 
ordered  after  WO 
start) _ 


Time  from  Mtl 
Complete  to  Work 
start 


number  of  items  added  after  BOM  firmed  &  WO  start  xl00% 


(number  items  ordered)] 


[(Material  Complete  date  -  first  Labor  Charged  date) 
for  all  WO]/  number  of  WOs 


Look  for 

increasing 

number  of 

material 

complete 

WOs. 


Look  for 
increasing 
frequency  of 
WOs  in 
extended  time 
categories. 


LL:  60% 
BL:  80% 
UL:  100% 


LL:  base 
established 
BL:  command 
average 
UL:  base 
established 


LL:  command 

average 

BL:  command 

average 

UL:  command 

average 


Look  for 
explanation  of 
increasing 
residue  as 
potential  poor 
planning  or 
stopped  jobs. 


LL:  0% 
BL:  10% 
UL:  20% 


LL:  base 
established 
BL:  base 
established 
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Inventory 
Accuracy  Rate 
(randomly 
selected  items) 


Inventory 

Unavailability 


number  of  incorrect  inventory  counts 


number  of  sampled  inventory  items 


number  of  items  not  provided  for  jobs  when  inventory 
in  mgt  system  stated  should  have  item 


Planning  Backlog  number  of  WOs  in  planning 


Facility  Surveys 
Completed 


Labor  Utilization 
broken  down  by 
work  category 
(emerg,  urgent, 
training,  etc.)  for 
each  Zone 


Availability  Rate 


Total  WOs  by 
location  in 
process 


Personnel 


Self-Help 

Customers 


Self-Help 

Expenditures 


number  of  facility  surveys  comnleted  x  100% 


number  of  facility  surveys  scheduled 


training  %  =  hrs  formal  training  (LUCs  20  &  32)/total 
direct  hrs 

RWP  %  =  hrs  RWP  (LUC  11)/  total  direct  hrs 
DSW  %  =  hrs  on  DSW  (LUCs  12,  14,  15,  16)/  total 
direct  hours 

Operations  %  =  hrs  operations  (LUC  19)/  total  direct 
hours 

Infrastructure  WO  %  =  hrs  spent  on  infrastructure  W/Os 
(LUCs  15  &  18)/  total  direct  hours 
Customer  WO  %  =  hrs  customer  WOs/  total  direct 
hours 

100  =  sum  of  all  above 


number  of  direct  labor  hours  bv  month  x  100% 


number  of  total  labor  hours 


number  of  WO  in  each  work  process  step  (planning, 
mat  cntl,  scheduling,  etc.) 


number  of  nersonnel  authorized  x  100% 


number  of  personnel  assigned 


number  of  customers  vs.  previous  FY 


targeted  $  self-help  expenditures  vs.  actual  $  self-help 


UL:  base 
established 


LL:  0 
BL:  5% 
UL:  10% 


LL:  0 
BL:  base 
established 
UL:  base 
established 


LL:  base 
established 
BL:  base 
established 
UL:  base 
established 


LL:  80% 
BL:  90% 
UL:  100% 


Look  for 
explanation  of 
spikes  in  any 
category. 

Base  should 
establish 
acceptable 
percentages  of 
each  category. 


LL:  7% 
BL:  85% 
UL:  95% 


Look  at 
backlogged 
areas  for 
explanation  or 
potential 
problems. 


LL:  80% 
BL:  90% 
UL:  100% 


Look  for 
explanation  of 
increasing  or 
decreasing 
trends. 


Look  at  trends 
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M-36 

Project  Review 
Turnaround 

number  of  projects  reviewed  on  time  vs.  late 

M-37 

WO/Completed 
Projects  awaiting 
As -Built  Update 

number  of  WOs  completed  requiring  as -built  updates 

M-38 

EMCS  Reliability 

number  of  hours  EMCS  svstems  onerational  x  100% 

number  of  hours  in  period 

M-39 

Operational 

Systems 

Fire  panels 
Transceivers 
Security  systems 

[number  of  systems  not  fully  operational  (based  on  open 
WO)]  /  [Total  number  of  systems] 

M-40 

Alarm  Systems 
WOs 

Emergency 

Urgent 

Routine 

number  of  WOs  vs.  number  of  delinquent  WOs 

M-41 

Roofing 

number  of  roofs  identified  as  bad  vs.  number  completed 
vs.  number  under  contract 

M-42 

Utility  Outages 
Power 

Water 

Gas 

Sewage 

number  of  outages  scheduled  vs.  other  cause  (weather, 

AF  breakage,  system  failure,  contractor  breakage)  - 
break  out  by  other  causes 

M-43 

Generator 

Operatability 

[(number  of  units  x  number  of  days  in  month)  -  number 
of  days  generators  out  of  service]/[n  umber  of  units  x 
number  of  days  in  month] 

M-44 

Aircraft  Arrest 

number  of  eneaeements 

System 

Operatability 

number  of  attempts 

LL:  80% 
BL:  90% 
UL:  100% 


LL:  80% 
BL:  90% 
UL:  100% 


Look  at  trends 
in 

delinquency 


Look  at  trends 
in  bad  roofs 
not  complete 
or  under 
contract. 


Look  at 
increasing 
trend  in  other 
cause  outages. 
May  reflect 
poorly 
updated  as- 
builts,  need 
for  new 
systems,  etc. 


LL:  80% 
BL:  90% 
UL:  100% 


LL:  90% 
BL:  95% 
UL:  100% 


130 


M-45 

Utility  Breaks 
Water 

Sewer 

Gas 

number  of  utility  breaks  by  utility 

Look  at  trends 
for  possible 
systems  that 
need  replacing 
or  increased 
maintenance. 

M-46 

Utility  Breaks 
Repaired  by 
(Gov’t, 
contractor) 

Water 

Sewer 

Gas 

number  of  breaks  repaired  source 

Increasing 

contractor 

breaks 
indicates 
potential  poor 
as-builts. 

M-47 

Appliance  Backup 
Stock 

number  of  backup  per  appliance  type 

Provides 
guidance  on 
when  backups 
should  be 
ordered. 

(Operations  Metrics,  2001,  n.  pag.) 
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Appendix  C.  Standard  and  Metric  Lines 


Appendix  C  contains  the  list  of  161  standards  and  metrics  specific  to  Operations  Flight 
used  in  this  research  effort.  The  standards  and  metrics  are  from  PWS  and  QASP 
documents  from  eight  Air  Force  Bases.  They  are  in  no  particular  order.  The  standards 
and  metrics  are  verbatim  from  the  data  that  was  found  in  the  documents.  Many  of  the 
acronyms  are  not  defined  because  they  did  not  impact  the  design  of  the  metric. 


Notes: 

IQL=10%  (15%):  Indefinite  Quality  Level;  first  variable  is  the  percentage  error  the  service  provider  is  held 
to  after  the  first  month  of  service.  The  second  variable  is  the  percentage  error  for  the  first  month  of  service. 

Any  portion  of  the  information  left  blank  in  the  documents,  the  corresponding  field  (e.g.,  threshold  or 
surveillance)  was  left  blank  in  the  appendices. 

Lot  is  the  number  that  will  be  sampled  in  the  given  time  period. 

Sample  size  is  the  number  that  will  be  reviewed  in  the  given  time  period. 

PR  =  0  (1):  Performance  Requirement;  first  variable  is  the  number  of  errors  the  service  provider  is  held  to 
after  the  first  month  of  service.  The  second  variable  is  the  number  of  errors  for  the  first  month  of  service. 
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Monthly  periodic 
review  of  operations 

and  repair  actions  1.5,  Provides  reliable,  cost  effective 
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Provide  Material  Control  Support 
all  CE  functions  Items  are 
requisitioned  and  stored  lAW  AFM 
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Storage  tanks,  valves, 

appurtenances,  and  piping  are  1.5,  Provides  reliable,  cost  effective 

inspected  and  maintained  lAW  0  DEFECTS  ALLOWED.  Lot  is  ui  i  utilities  to  meet  readiness  needs.  Operational 
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identified  in  PWS.  maintain  quality  of  life. 
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Measure  the  unit  cost  per  occupant 
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Measure  the  percentage  of 
unscheduled  work  performed  on  the 

installation  as  a  portion  of  total  1 .7,  Establishes  quality  standards  and 
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Measure  the  average  schedule 
variance.  Schedule  variance  is  a 


162 


Measure  the  number  of  times  the 

Service  Provider  was  not  timely  on  1 .7,  Establishes  quality  standards  and 
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Measure/track  the  overall  cost  of 
utilities  per  square  foot  of 
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Measure  the  total  annual  British 
Thermal  Unit  (BTU)  consumption 
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Provide  a  trend  analysis  for  the  total 
maintenance  cost  based  on 
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'proactive". 
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Measure  the  number  of 
unscheduled  outages  of  all  critical 


168 


Measure  the  mean  time  between 
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action  is  taken). 


The  service  provider  will  be 
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Appendix  D.  1.1  Operations  Flight  Functions 


Appendix  D  contains  each  of  the  lines  classified  as  1.1,  Operate,  maintain,  repair, 
construct,  and  demolish  real  property  and  RPIE  to  accomplish  the  mission  in  most  timely 
and  economical  manner  from  Appendix  C.  The  primary  line  evaluation  table  associated 
with  the  metric  and  standard  is  listed  below  its  respective  metric  and  standard  line. 
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(Reduced  personal  influence 

or  judgment) _ 

High  Impact?  (Affect  Quality  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of  life,  and  successful  completion 
of  Life,  Mission,  or  Customer) _ of  the  mission _ 
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(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of  life, 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ and  successful  completion  of  the  mission _ 


Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 

Maintain  real  property  to  a  2  Defects.  Lot  is  number  of  Emergency,  1.1,  Operate,  maintain,  repair,  construct,  and 

standard  commensurate  with  Urgent,  Routine,  &  Minor  Construction  Random  demolish  real  property  and  RPIE  to 

design  criteria  and  accepted  unplanned  work  orders  completed  in  a  Sampling  accomplish  the  mission  in  most  timely  and  ^  ^ 

industry  standards.  month.  economical  manner 
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Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 

Provide  economical  1  Defect.  Lot  is  number  of  1.1,  Operate,  maintain,  repair,  construct,  and 

maintenance  and  planned  work  orders  Checklist  demolish  real  property  and  RPIE  to  accomplish  the  Finance/Budget 

repair  of  facilities.  completed  in  a  month.  mission  in  most  timely  and  economical  manner 
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Fligh  Impact?  Yes  Flas  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of  life,  and 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ successful  completion  of  the  mission _ 
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(Reduced  personal  influence  or  judgment) _ and  repair  are  done  I  AW  PWS 

High  Impact?  Yes  Affects  completion  of  mission 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ 


ID  Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 

Infrastructure  Programs:  Manage,  Less  than  5%  Periodic  review  of  contractor  1.1,  Operate,  maintain,  repair,  construct, 

61  maintain,  and  implement  infrastructure  deviation,  measured  interface  with  customers;  and  demolish  real  property  and  RPIE  to  xjjne/Schedule 
programs  within  30  days  lAW  the  semi  annually  review  of  infrastructure  accomplish  the  mission  in  most  timely 
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Quantifiable?  No  Due  to  confusion  between  the  threshold  and  surveillance.  Required  personal  judgment  of 

(Reduced  personal  influence  or  judgment) _ what  is  desired. _ 

High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of  life,  and 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ successful  completion  of  the  mission _ 
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Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 

DSW  Completion:  Properly  executed  Daily  monitoring  of  1.1,  Operate,  maintain,  repair,  construct,  and 

Complete  DSW’s  lAW  90%  of  the  time,  on  DSW  accomplishments,  demolish  real  property  and  RPIE  to  accomplish  the  Quality 

API  32-1004v3  time  frame.  a  weekly  basis.  customer  complaints  mission  in  most  timely  and  economical  manner 
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ID  Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 

Maintain  base  real  property,  accountable  real  100%  of  taskings  shall  Review  monthly  1.1,  Operate,  maintain,  repair,  construct, 

73  property  installed  equipment  (RPIE),  and  civil  meet  established  suspenses  metric(s)  and  and  demolish  real  property  and  RPIE  to  Operational 
engineering  responsibility  non-RPIE  items  (e.g.  with  no  more  than  5%  customer  accomplish  the  mission  in  most  timely  Service  Levels 

dorm  furniture,  microwaves,  appliances,  etc.).  requiring  correction.  complaint.  and  economical  manner 
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ID  Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 

Manage  workload  within  No  facility/infrastructure  related  1.1,  Operate,  maintain,  repair,  construct,  and 

75  budgetary  constraints  and  in  discrepancy  shall  prevent  Alert  Customer  demolish  real  property  and  RPIE  to  accomplish  .  , 

keeping  with  engineering  aircraft  ability  to  depart  with  1 5  complaint.  the  mission  in  most  timely  and  economical  ° 

_ life  cycle  cost  rationale. _ minute  notice. _ manner _ 
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(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of  life,  and 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ successful  completion  of  the  mission _ 
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(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Affect  mission  capability 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ 


"£ 

0 

_o 

’-P 

(D 

> 

V 

H-l 

?3 

(D 

i-i 

<D 

O 

0. 

'> 

O 

0 

(D 

m 

COD 


(D 

T3 

0 

C3 


-s  ^ 


■-Hi 


TJ 

PJ  C 

HH  c?3 

Oh 

II 

t  o 

a,  » 

p 


w  c 
£  '£ 
®  "a 
§  S 
U  g 


<Hh 

O 

©N  . 

O  ■ 

o 


cd 

-C 


0 

*«  i-. 

C  W 

!  s 

O  40 
S  O 

’3  >. 

O"  T3 

<D  ITS 
^  (D 
GO  i-> 

S  tu 
S  -r> 


<  3 


c^ 

00 


185 


s  “  S 

2  M  '5  CN 

2  w  a,  II 

-c  u  ' 

M  rr,  Qi 

^  (N  ^  CL- 

s-  li  o  ^ 

■*  ,  i-  ■£ 

^  .a|  g 

^  I  s 

q  c 


^  g  ^ 

3:  .S 

>.  CT^  -O 

pr 

§  fD  ’-g 

o  o  c: 

^  03  T3 

frt  Ch  '”" 

^  ^ 

"  b 

!-  --I  03 

O  =2  73 

g  ^  £ 

+3  C3  ^ 

S  s  a 

o  g  ^ 

^  '3  ^ 


"S  •£  o- 
S  g  “  ■ 

ca  S  < 


c/5  c/5  c/5 

:>  <i)  D  o  o 

>H  >H  >H  Z 


■S  ii 

C^*  1-^  ^ 

>^0  .2 
g  CCJ  ^ 
C'"  ^  ^ 

g  I 

S3  O  ^=H  O 

2  ^  i 

c/)  Ph  D  O 


O  t3 

■s-  “  o  S 

Q>  Lh  ^  c<3 

"  (u  g  a 
o  S  -35 

•12  J-*  O 

S  .2 
■S  13  .a  S 

s  a  s  o 
s  ^  fi 

£  ;a  •£  g 

g  "o  J3 

a  6  ;2 

O  -U  & 
-'go 
-  §  g 

CO 


-c3  >  . 

.t3  O  ^ 

>“>  co  C^ 

"H-  a  S3 

S3  <L»  •’^ 

o  S  "2 

O  <D  .25 

^  U-i  ^ 

S  O'  ^ 

■«  2  -a 


o  1  ”2 
2  || 
PI  u  g 
o  >  g 
o  03  -s 
u  <u 
—  41 


■g  a  c  g 
o  a  g  g 
S  <  g  CQ 


C/5  C/5  C/5 

^  <L>  <L>  (1>  O 


43  - 
±5  c3  *33 

32  f>'  ’-rt  3S 

S  >>  ^  g  43 

^  2  t3  jC  .2 

TS  c^'  P*  ^  42 

u  2  2  2  '-S  — 

53  o  p  2  ^  g 

f32  (/2  or  py  S 

2^  ^  u2  S 


187 


High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering, 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ quality  of  life,  and  successful  completion  of  the  mission _ 
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(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of 

(Affect  Quality  of  Life.  Mission,  or  Customer) _ life,  and  successful  completion  of  the  mission  (installation  security) _ 
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High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ life,  and  successful  completion  of  the  mission _ 
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(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of 

(Affect  Quality  of  Life.  Mission,  or  Customer) _ life,  and  successful  completion  of  the  mission _ 
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Appendix  E.  1.2  Operations  Flight  Functions 


Appendix  E  contains  each  of  the  lines  classified  as  1 .2,  Provides  trained  personnel  and 
technical  expertise  to  support  operations  worldwide,  from  Appendix  C.  The  primary  line 
evaluation  table  associated  with  the  metric  and  standard  is  listed  below  its  respective 
metric  and  standard  line.  . 
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(Affect  Quality  of  Life,  Mission,  or  Customer) 
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High  Impact/  Y es  Has  signihcant  impact  upon  the  customers  or  the  civil  engineering, 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ quality  of  life,  and  successful  completion  of  the  mission _ 
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(Reduced  personal  influence  or  judgment) _ objective  is  open  to  debate  if  the  meetings  are  not  clearly  listed _ 

High  Impact?  No  This  is  an  administrative  issue  that  should  not  impact  quality  of  life,  customers, 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ or  mission _ 
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Quantifiable?  Yes  Based  upon  missing  agendas  at  meetings 

(Reduced  personal  influence  or  judgment) _ 

High  Impact?  No  This  is  an  administrative  issue  that  should  not  impact  quality  of  life,  customers, 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ or  mission _ 


ID  Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 

All  reports  and  schedules  are  1  DEFECT  ALLOWED.  Lot  is  total  #  of  1 .2,  Provides  trained  personnel  and 

38  provided  in  the  form/format  reports  and  schedules  required  each  100%  Inspection  technical  expertise  to  support  Quality 
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Quantifiable?  Yes  Based  upon  the  number  of  facilities  not  having  maintenance  folders 

(Reduced  personal  influence  or  judgment) _ 

High  Impact?  No  This  is  an  administrative  issue  that  should  not  impact  quality  of  life, 

(Affect  Quality  of  Life,  Mission,  or  Customer)  customers,  or  mission 
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High  Impact?  No  This  is  an  administrative  issue  that  should  not  impact  quality  of  life,  customers,  or 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ mission _ 
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High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering, 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ quality  of  life,  and  successful  completion  of  the  mission _ 


ft  <D  . 

J3  X3 

.£f  ^  ■- 


5-  ^  P 
cr  cd  ^ 

.H  S) 

9  •'g  T3 
^  O  ^ 

c/3  -»-< 

C/3  ^  C 

“  ^  M 

O  ^ 

^  Si 

^ 


(11  C/3  ^  <1) 

^  T3  -7:  i= 

"S  O  c  ^ 

0  o 

:2  ^  >5 

>  cc:  2  H 


c/3  c/3  c/3 

O 

2; 


ti  "::5 

2  C^*  ft  3 

S  >1  C  'J 

^  05c 

Ts  (^-  c  « 

g  S  §  g  '3 

«  a  S?  '2  0 


u  C  2 

t-  QJ  3 

^  s5  £ 
1  §  i- 
■5  I  2  ■£ 

(a>  C3  rv  -ti 

J=  a  ^  ^ 

H  ^  2  t; 

g. 

(D 

c^  S3  O 
Cl( 

"^T’  »-<  ft 
ft  o  ;j:s 

(73  .j- 

ft 

T3  c3 


Q 

B  00 


ft  o  0^  ^ 

o  o  ^  w 

^  ei  Z  H 


VM  ^ 

>^0  .2 
£  c3  ft 


(^  £  5 


as 

0 

O 

C-.  u  <>• 

“  p-  2 

Xa  ^  as 

13  T3  cx 

S  8  6 

I  •€  S 

Sum 

1^  ^  m 


203 


(Affect  Quality  of  Life,  Mission,  or  Customer) 
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(Reduced  personal  influence  or  judgment) _ 

High  Impact?  No  This  is  an  administrative  issue  that  should  not  impact  quality  of  life, 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ customers,  or  mission _ 
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High  Impact?  No  This  is  an  administrative  issue  that  should  not  impact  quality  of  life, 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ customers,  or  mission _ 
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Appendix  F.  1.3  Operations  Flight  Functions 


Appendix  F  contains  each  of  the  lines  classified  as  1.3,  Maintain  capability  to  respond  to 
and  eliminate  any  emergency  condition  24  hours  a  day,  from  Appendix  C.  The  primary 
line  evaluation  table  associated  with  the  metric  and  standard  is  listed  below  its  respective 
metric  and  standard  line. 
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ID  Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 

Emergency  Work  Requests:  30  minutes  (duty  ^  ofthe  ^^^^ords  Review  or  1.3,  Maintains  capability  to  respond  to  and 
5  hours)/ 1  hour  (non-duty  hours);  completed  Customer  Contact,  at  eliminate  any  emergency  condition  24  hours  a  Time/Schedule 

(safed)  in  24  hours.  least  1/week  day 
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Quantifiable?  Yes  Based  upon  the  number  of  emergency  work  requests 

(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering, 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ quality  of  life,  and  successful  completion  of  the  mission _ 
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Appendix  G.  1.4  Operations  Flight  Functions 


Appendix  G  contains  each  of  the  lines  classified  as  1.4,  Conducts  all  activities  in  compliance 
with  applicable  safety  laws,  codes,  and  directives,  from  Appendix  C.  The  primary  line 
evaluation  table  associated  with  the  metric  and  standard  is  listed  below  its  respective  metric 
and  standard  line. 
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_ Objective _ Threshold _ Surveillance  Operations  Flight  Function  Gauge  Cluster 

Material  Safety  Data  Sheets  (MSDS).  Upon  receipt  of  MSDS’s  are  available  1.4,  Conducts  all  activities  in 

material,  the  contractor  shall  establish  and  maintain  for  all  material  within  Periodic  review  of  compliance  with  applicable  Work  Product 
the  required  MSDS  for  materials  and  forward  to  the  1 5  working  days  of  contractor  records  safety  laws,  codes,  and  Delivered 

appropriate  section.  receipt.  directives 
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Appendix  H.  1.5  Operations  Flight  Functions 


Appendix  H  contains  each  of  the  lines  classified  as  1 .5,  Provides  reliable,  cost  effective 
utilities  to  meet  readiness  needs,  satisfy  installation  needs,  and  maintain  quality  of  life, 
from  Appendix  C.  The  primary  line  evaluation  table  associated  with  the  metric  and 
standard  is  listed  below  its  respective  metric  and  standard  line. 
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(Not  difficult  to  understand) _ 

Quantifiable?  No  Because  the  objective  safe  start  up  cannot  be  evaluated  by  time 

(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering, 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ quality  of  life,  and  successful  completion  of  the  mission _ 
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Quantifiable?  (Reduced  personal  influence  or  judgment) _ No _ Because  the  objective  of  testing  cannot  be  evaluated  by  time _ 

High  Impact?  (Affect  Quality  of  Life,  Mission,  or  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of 

Customer) _ life,  and  successful  completion  of  the  mission _ 
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High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering, 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ quality  of  life,  and  successful  completion  of  the  mission _ 
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(Affect  Quality  of  Life,  Mission,  or  Customer) _ | _ |  life,  and  successful  completion  of  the  mission 
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Quantifiable?  No  The  amount  of  leaks  &  corrosion  is  not  quantifiable,  it  is  subjective  to  evaluator 

(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ life,  and  successful  completion  of  the  mission _ 
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Quantiriable  /  No  Because  the  area  is  too  broad 

(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Affects  quality  of  life  and  mission 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ 
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High  Impact  /  Y  es  Has  signihcant  impact  upon  the  customers  or  the  civil  engineering,  quality  o: 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ life,  and  successful  completion  of  the  mission _ 


—  w 

I  -S  ^ 

|_g  II 
■S  t3  cii 

pq  g  ^ 


N  55  V-. 

^  ^  Oh 

+e  "O  t/2 

►— 1  X  ^ 
o  CS 


C3  ^  . 

^ 

'S  -2  ^ 

^  C  0-1 

>>  ^  c 

■a  £  :r 

^  (D  "O 
G  O 

R  =5  ^ 

=P  p 
^  S, 
2  "O 

s 

9  >^  -2 
S  u  s 

8  ““ 

O  fH  g 

^'S - 

H  Q 


~0  Ti 

^  O  c  ~o 
C3  p  ^  — 

H  Gi  S  < 


CO  CO  tzi 

>,  <U  D  (U  o 


CfH 

O 

a 

•'G 

3 

G 

cr 

t>S 

^G 

'C 

(U 

(U 

^G 

'5b 

G 

(U 

1— H 

■> 

'o 

(U 

C4-, 

G 

o 

_o 

CO 

CO 

u 

CO 

a> 

a 

B 

o 

a> 

CO 

^3 

O 

Cl-H 

o 

(U 

G 

O 

G 

O 

a> 

iX 

o. 

G 

p 

o 

o 

C5 

(L> 

X 

_ 

c 

G 

G 

<G 

CO 

CO 

G 

a> 

C5 

o 

o 

o 

3 

G 

CO 

'g 

"G 

.£? 

G 

'w 

cs 

CO 

cs 

1 

n: 

^  CO 

-5  ii  c>*  «-. 

O-.  -g  3  22  M 

I  I  ^  c^  ^  ' 

T3  C^.  C  ^  C  c  o 

'“'o  §  g^ll 


<>*  3 

CO  2 

ts  J  ■ 

GO  XJ  ,(l> 

-  d  ^ 

CO  C«  1— ( 
XJ  r,  C4-( 

a>  M  o 

(U  T3  ^ 
c:  <i>  ^ 
a>  .X 


’rt 

X)  S  ^ 
£  3  a 
§^o 
T3  -g  II 
S  2  Oi 

I  §“ 

■B-^  s 

CO  C3  c 

W  c«  o 
—  -o  g 
ro  22  S 

II  •€  B 

MB  ^ 


>.  CT*  X) 
—  a>  a> 

Cl  w  ,— 

s  “  ^ 

o  £  c 
o  (u  a> 
=  «  .'2 
>>  r/i 

xi  C/J  -S 

sfl 

ill 

cJ  h- 3  (u 


M-H 

H  .b  < 

<  ^ 


45  —  G 

CO  ir  ■£  •“  O 

^  O  n  c3  -r 

“  ^  I  5  1 

H  02  S  <  £ 


fli>  CO  CO  CO 

>(  O  O  0)  o 

>-  Z 


so  S  S 

in  2  O' 

^  n  (u  3 


C^.  I  3 
>>a^3 
n  2  fc  iH 

o  pr;  ^ 


•n  o  S 

^  rv 


229 


(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ life,  and  successful  completion  of  the  mission _ 
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High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ life,  and  successful  completion  of  the  mission _ 
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(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ life,  and  successful  completion  of  the  mission _ 
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(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ life,  and  successful  completion  of  the  mission _ 
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High  Impact  /  Y es  Arrects  quality  or  lire  and  mission 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ 


ID  Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 
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Understandable?  (Not  difficult  to  understand) _ Yes _ All  utility  costs  associated  with  a  building 

Quantifiable?  (Reduced  personal  influence  or  judgment) _ Yes _ Based  upon  the  cost  of  utilities  for  a  buildin; 

High  Impact?  (Affect  Quality  of  Life,  Mission,  or  Customer)  Yes _ Affects  customers  and  mission _ 
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Appendix  I.  1.6  Operations  Flight  Functions 


Appendix  I  contains  each  of  the  lines  classified  as  1 .6,  Provides  base  support  services, 
from  Appendix  C.  The  primary  line  evaluation  table  associated  with  the  metric  and 
standard  is  listed  below  its  respective  metric  and  standard  line. 
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(Affect  Quality  of  life.  Mission,  or  Customer) 
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(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of  life,  and 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ successful  completion  of  the  mission _ 
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(Not  difficult  to  understand) _ 

Quantifiable?  Yes  Based  upon  recorded  schedules  of  refuse  collectors 

(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Affects  quality  oflife  and  customers 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ 


Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 
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(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Has  significant  impact  upon  the  customers  of  the  civil  engineering,  quality  of  life,  and 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ successful  completion  of  the  mission _ 
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High  Impact?  Yes  Affects  mission 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ 
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Appendix  J.  1.7  Operations  Flight  Functions 


Appendix  J  contains  each  of  the  lines  classified  as  1 .7,  Establishes  quality  standards  and 
feedback  mechanisms  to  assess  performance  in  meeting  mission  requirements  and  customer's 
needs,  from  Appendix  C.  The  primary  line  evaluation  table  associated  with  the  metric  and 
standard  is  listed  below  its  respective  metric  and  standard  line. 


247 


C3 

a 


0 

c/3  C/3 

73  ^ 

U.  c/3 

P3  ^ 
n3  c3 

g  o 

+-i 

4-* 

C/3  !/3 


s 

^  cd 

^  O 
c/3  (L) 

^  s 
.2  >! 
3  ^ 

C3  o 
X3 
nj  <U 

r-- 


0 

.2  Si 

c/3 


c  S 

S  o 

'o  tJo 

<U  ^ 

S 

S  c= 

cd 

S  ^ 
§  g 


0 

CT 


2  ^ 

■> 


c/3 


U 


^  2 

-  s  3  a 

*s^  "  ^  cd 

■O  u  C<-(  u 

-O'" 


g  “ 
& 


O 


0 

O 

O 


CJ 

O  a; 
\®  £ 


c/3  .£  — 


73 

. 

-2 

"3 

0 

73 

a 

0) 

43 

73 

o 

QJ 

c/3 

> 

C/! 

O 

Cd 

i-* 

a 

-10 

CIh 

CD 

cd 

O 

a 

=3 

"O 

Ph 

0 

0) 

O 

-S 

CD 

QD 

Cd 

< 

c/3 

c: 

73 

.2 

CD 

CD 

O 

QD 

1) 

a 

5-  ^ 

0)  CD 

cx 

o 


cd 

0  . 
a 

CD 

> 


w 


^  'S 

bD 

o  tB 

O 

i-< 

C/l  S 

a 

ao  d 

■— * 

■§3 

o 

i-i 

S  g 

c 

o 

g  ^ 

U 

C 

Ph 


^  <! 


u 

a> 

-2 

CA 

5 

-o 

o 

0 

o 

4> 

C/0 

00 

Id 

F3 

0 

9S 

o 

H 

bo 

i  i 

flD  B 

s 

^  -  CM 

"U  .S 

o 

CD 

s 

0  O 

^  n  B 

^  0  O 
^  cd  •Ti 

9 

5-(  ^  CO 

cd  E 

73  C  o 

JZ 

3f 

S  c2  t3 
i5  t-H  cl 

CO  ^  cd 

b 

^  S'  S 

CM 

S 

CO  0 

<D  <D 

1  ^  S 

<R 

cr  cd  <D 

OD  -B  3 

dD 

c/3  ' 

a. 

.22  c  ^ 

O 

^  .22  c 
iS  c  o 

c/3  Cd 

W  .g  M 

C-:  2  £ 

T—  « 

CD 

<4-< 

s 

o  o 

0^1 
.2  CD  ^ 

> 

■>  Q. 

la 

CP 

3 

i/) 

^  CO 

2 

£  c  > 

2 

■"  T3  £ 

JS 

CM 

2  1)  e 
-C  L-  5 

d> 

't:  22  >> 

4S 

®  2  S 

H 

S  c 

v~>  O 

S 

d  -o 

Cd 

(M  CD  413 

CD  •::;  Cd 

C3  CD 

^  CD 

CJ 

w  ^  <  jj 

s'  u  ■£  'o 

o 

d> 

ci  cd  (D 

15* 

^  ^  "b  -C 

o 

III" 

d 

tS  a  cr 
o  S  g 

u  1  i: 

'  S  tS  I 

1 

cu 

o 

o^ 

»/~i 

248 


250 


frequency/ _ Yes _ Monthly _ 

Understandable?  (Not  difficult  to  understand)  Yes  Obtain  costs  per  customer  of  all  facilities  used  or  maintained  by  the  service 

_ provider _ 

Quantifiable?  (Reduced  personal  influence  or  judgment) _ Yes _ The  objective  clearly  defines  what  is  included;  reduces  personal  judgment 

High  Impact?  (Affect  Quality  of  Life,  Mission,  or  Customer)  Yes _ Affects  quality  of  life  and  customers _ 
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Understandable?  (Not  difficult  to  understand) _ Yes _ Obtain  the  difference  between  estimates  and  actual  time  to  complete 

Quantifiable?  (Reduced  personal  influence  or  judgment)  Yes  Based  upon  the  schedule  data  found  in  the  records;  reduces  personal 

_ judgment _ 

High  Impact?  Yes  Affects  customers  and  quality  of  life 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ 


ID  Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 

Report  the  average  cycle  time  it  takes  the  1.7,  Establishes  quality  standards  and  feedback 

146  Service  Provider  to  provide  reasonable  Monthly  mechanisms  to  assess  performance  in  meeting  Time/Schedule 
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Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 

Measure  the  mean  time  between  failures  of  facility  components,  systems,  and  real 
property  installed  equipment  (RPIE).  The  metric  will  serve  as  an  indicator  of 
facility  components,  systems,  and  RPIE  that  may  be  reaching  their  service  life  or 
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Appendix  K.  1.8  Operations  Flight  Functions 


Appendix  K  contains  each  of  the  lines  classified  as  1.8,  Establishes  a  system  to  provide 
customers  the  capability  to  accomplish  work  requirements  using  their  own  resources,  from 
Appendix  C.  The  primary  line  evaluation  table  associated  with  the  metric  and  standard  is 
listed  below  its  respective  metric  and  standard  line. 
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Objective  Threshold  Surveillance  Operations  Flight  Funcfion  Gauge  Cluster 

Establish,  maintain  and  2  Defects.  Lot  is  the  number  Customer  1.8,  Establishes  a  system  to  provide  customers  the  capability  Operational 
operate  a  Self  Elelp  store  of  customers  serviced  monthly.  Complaint  to  accomplish  work  requirements  using  their  own  resources  Service  Levels 
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Quantifiable?  No  Because  the  area  is  too  broad 

(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Affects  customers 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ 


Appendix  L.  1.10  Operations  Flight  Functions 


Appendix  L  contains  each  of  the  lines  classified  as  1.10,  Effectively  allocates  in-service 
resources  to  meet  mission  and  customers'  needs,  from  Appendix  C.  The  primary  line 
evaluation  table  associated  with  the  metric  and  standard  is  listed  below  its  respective  metric 
and  standard  line. 
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High  Impact?  No  This  is  an  administrative  issue  that  should  not  impact  quality  of  life, 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ customers,  or  mission _ 
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(Not  difficult  to  understand) _ breakdown,  thus  the  service  provider  may  have  to  be  reactive. _ 

Quantifiable?(Reduced  personal  influence  or  judgment) _ No _ Personal  judgment  influences  what  the  percentage  represents _ 

High  Impact?  No  This  is  an  administrative  issue  that  should  not  impact  quality  of  life, 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ customers,  or  mission _ 


Appendix  M.  1.12  Operations  Flight  Functions 


Appendix  M  contains  each  of  the  lines  classified  as  1.12,  Maintains  a  time  and  material 
accounting  system  to  collect  and  report  the  cost  of  doing  business,  from  Appendix  C.  The 
primary  line  evaluation  table  associated  with  the  metric  and  standard  is  listed  below  its 
respective  metric  and  standard  line. 
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Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 

lalysis  of  all  work  orders  1  Defect.  Lot  is  number  of  1.12,  Maintains  a  time  and  material 

greater  difference  between  completed,  planned  work  orders  Checklist  accounting  system  to  collect  and  Time/Schedule 

md  estimated  hours.  for  the  month.  report  the  cost  of  doing  business 
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(Affect  Quality  of  Life,  Mission,  or  Customer) _ _ |  life,  customers,  or  mission 
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Quantifiable?  No  Personal  judgment  will  influence  the  amount  required  to  have  on  hand 

(Reduced  personal  influence  or  judgment) _ 

High  Impact?  Yes  Affect  customers 

(Affect  Quality  of  Life,  Mission,  or  Customer) _ 


Appendix  N.  1.13  Operations  Flight  Functions 


Appendix  N  contains  each  of  the  lines  classified  as  1.13,  Provides  effective  logistics 
support,  from  Appendix  C.  The  primary  line  evaluation  table  associated  with  the  metric 
and  standard  is  listed  below  its  respective  metric  and  standard  line. 
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Appendix  O.  Lines  That  Passed  the  Primary  Evaluation 


Appendix  O  contains  all  of  the  lines  that  passed  the  primary  evaluation,  regardless  of 
their  Operations  Flight  Function  classification,  from  Appendix  C.  The  secondary  line 
evaluation  table  associated  with  the  metric  and  standard  is  listed  below  its  respective 
metric  and  standard  line.  The  information  contained  within  the  secondary  evaluation 
tables  is  based  upon  information  received  from  the  respective  base  personnel. 
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_ Objective _ Threshold _ Surveillance _ Operations  Flight  Function _ Gauge  Cluster 

DSW  Completion:  Timely  resolution  Daily  monitoring  of  1.1,  Operate,  maintain,  repair,  construct,  and 

Complete  DSW’s  lAW  API  90%  of  the  time,  on  DSW  accomplishments,  demolish  real  property  and  RPIE  to  accomplish  the  Time/Schedule 
32-1004v3  time  frame.  a  weekly  basis.  customer  complaints  mission  in  most  timely  and  economical  manner 
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Proven?  Yes  If  the  work  request  is  not  completed  in  specified  time,  the  QAE  questions  the 

(Has  shown  demonstrated  results) _ service  provider  and  initiates  closure  as  close  to  the  specified  time  as  possible 
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Proven?(Has  shown  demonstrated  results)  Yes  Contractor  completes  work  in  the  next  month  for  work  not  completed  in  the 

expected  month. 
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30  standard  commensurate  with  Urgent,  Routine,  &  Minor  Construction  Random  and  demolish  real  property  and  RPIE  to 
design  criteria  and  accepted  unplanned  work  orders  completed  in  a  Sampling  accomplish  the  mission  in  most  timely 
industry  standards.  month.  and  economical  manner 
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scheduled  monthly  System  mission  in  most  timely  and  economical  manner 


Objective  Threshold  Surveillance  Operations  Flight  Function  Gauge  Cluster 
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150  $5,958,108  on  electricity  in  FY98  and  had  a  gross  square  footage  (of  in  total  energy  Monthly  readiness  needs,  satisfy  Productivity 

buildings)  of  7,434,170.  $5,958,108/7,434,170  results  in  $0.80144/sf  for  costs  per  installation  needs,  and 

electricity.  The  metric  will  have  three  columns;  one  for  electricity,  one  for  square  foot.  maintain  quality  of  life. 

_ gas,  and  one  for  water. _ 


Objective  Threshold  Surveillance  Operations  Flight  Function 
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Measure  the  unit  cost  per  occupant  to  maintain  and 
operate  the  facilities  used  or  maintained  by  the  service 
provider  (SP).  Costs  shall  include  all  direct  and  indirect 
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Objective  ThresholdSurveillance  Operations  Flight  Function 
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Appendix  P.  Operations  Flight  Function  and  Gauge  Cluster  Classifications  of 
AFCESA  Operations  Flight  Metrics 


Appendix  P  contains  the  Operations  Flight  Function  and  Gauge  Cluster  classifications  of 
the  metrics  found  on  the  AFECSA  website  to  evaluate  the  Operations  Flight.  Each  of  the 
47  metrics  is  classified  and  listed  separately. 
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Objective  Threshold  Surveillance  Function  Gauge  Cluster 
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training  %  =  hrs  formal  training  (LUCs  20  &  32)/total 
direct  hrs 

RWP  %  =  hrs  RWP  (LUC  11)/  total  direct  hrs 
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Appendix  O.  Operations  Flight  Function  and  Gauge  Cluster  Classifications  of 

AFCESA  Template  Metrics 


Appendix  Q  contains  the  Operations  Flight  Function  and  Gauge  Cluster  classifications  of 
the  metrics  found  in  the  service  delivery  summary  of  the  Operations  Flight  PWS  template 
found  on  the  AFECSA  website.  Each  of  the  9  metrics  is  classified  and  listed  separately. 
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Objective  Threshold  Operations  Flight  Function  Gauge  Cluster 
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Provide  an  effective  design  program  and  construction  1.9,  Develops  and  annually  updates 

management  to  ensure  projects  are  designed  and  .  future  plans  for  major  work  _.  ,„  ,  ,  , 

.  ,  ,  100%  of  time  .  .  r-  Time/Schedule 

construction  work  are  completed  timely  and  within  requirements  (rooting,  pavements, 

budget.  and  protective  coating) 
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Appendix  R.  Primary  Evaluation  of  AFCESA  Operations  Flight  Metrics 


Appendix  R  contains  each  of  the  lines  from  Appendix  P  (46  AFCESA  Operations  Flight 
Metrics).  The  primary  line  evaluation  table  associated  with  the  metric  and  standard  is 
listed  below  its  respective  metric  and  standard  line. 
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Quantifiable?  (Reduced  personal  influence  or  judgment) _ Yes _ Number  of  work  orders  by  category _ 

High  Impact?  (Affect  Quality  of  Life,  Mission,  or  Customer)  No _ Administrative  issue  that  tracks  the  number  of  work  orders 


293 


Frequency? _ No _ No  frequency  listed _ 

Understandable?  (Not  difficult  to  understand) _ No _ There  should  not  be  a  lower  threshold  limit 

Quantifiable?  (Reduced  personal  influence  or  judgment) _ Yes _ Based  upon  the  time  to  complete  DSW _ 

Fligh  Impact?  (Affect  Quality  of  Life,  Mission,  or  Customer)  Yes _ Affects  customer,  mission,  and  quality  of  life 
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Understandable?  (Not  difficult  to  understand) _ Yes _ Tracks  number  of  work  orders  in  backlog _ 

Quantifiable?  (Reduced  personal  influence  or  judgment) _ Yes _ Based  upon  number  of  work  orders  received  and  completed 

High  Impact?  (Affect  Quality  of  Life,  Mission,  or  Customer)  No _ Administrative  issue  that  tracks  backlog _ 
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Affect  Quality  of  Life,  Mission,  or  Customer)  |  Yes  |  Affects  mission  and  customer 
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Quantifiable?  (Reduced  personal  influence  or  judgment) _ Yes _ Based  upon  the  number  of  unfinished  work  orders 

High  Impact?  (Affect  Quality  of  Life,  Mission,  or  Customer)  Yes _ Affects  customer,  mission,  and  quality  of  life 
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Understandable?  (Not  difficult  to  understand) _ No _ The  objective  is  confusing  because  material  complete  WO  is  not  defined 

Quantifiable?  (Reduced  personal  influence  or  judgment) _ No _ Based  upon  the  objective  classified  as  not  understandable _ 

High  Impact?  (Affect  Quality  of  Life,  Mission,  or  Customer)  No _ Administrative  issue  that  tracks  material  and  work  orders _ 
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Affect  Quality  of  Life,  Mission,  or  Customer)  |  No  |  Administrative  issue  that  tracks  bench  stock 
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High  Impact?  (Affect  Quality  of  Life,  Mission,  or  Customer)  |  No  |  Administrative  issue  that  tracks  material  and  work  order  completion  times 
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Affect  Quality  of  Life,  Mission,  or  Customer)  |  No _ |  Administrative  issue  that  tracks  the  items  missed  in  the  initial  material  order 
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Frequency? _ No _ No  frequency  listed _ 

Understandable?  (Not  difficult  to  understand) _ No _ Surveillance  not  well  defined _ 

Quantifiable?  (Reduced  personal  influence  or  judgment) _ No _ Based  upon  surveillance  classified  as  not  understandable 

High  Impact?  (Affect  Quality  of  Life,  Mission,  or  Customer)  No _ Administrative  issue  that  tracks  labor _ 
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Affect  Quality  of  Life,  Mission,  or  Customer)  |  No  |  Administrative  issue  that  tracks  breaks 
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Appendix  S.  Primary  Evaluation  of  AFCESA  Template  Metrics 


Appendix  S  contains  each  of  the  lines  from  Appendix  Q  (9  AFCESA  Service  Delivery 
Summary  Metrics  found  on  the  Operations  Flight  competitive  sourcing  template).  The 
primary  line  evaluatbn  table  associated  with  the  metric  and  standard  is  listed  below  its 
respective  metric  and  standard  line. 
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ID _ Objective _ Threshold _ Operations  Flight  Function _ Gauge  Cluster 

T-7  Provide  an  effective  design  program  and  construction  100%  of  time  1.9,  Develops  and  annually  updates  Time/Schedule 

management  to  ensure  projects  are  designed  and  future  plans  for  major  work 

construction  work  are  completed  timely  and  within  requirements  (roofing,  pavements, 

budget.  and  protective  coating) 
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_ Objective _ Threshold _ Operations  Flight  Function _ Gauge  Cluster 

Respond  to  contingencies  and  natural  disasters  during  100%  of  time  1.3,  Maintains  capability  to  respond  Time/Schedule 

normal  and  after  duty  hours  within  required  time  limits  to  and  eliminate  any  emergency 

condition  24  hours  a  day 
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